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1.  SUMMARY 

The  DARPA  META-II  program  goal  is  to  substantially  improve  the  design, 
manufacturing,  and  verification  of  complex  cyber-physical  systems,  and  particularly  defense  and 
aerospace  systems  such  as  ground  combat  vehicles,  airplanes,  and  rotorcraft.  The  program  is 
motivated  by  the  accelerating  complexity  of  systems.  DARPA  META  program  manager  Mr. 

Paul  Eremenko  describes  the  goal  as:  “to  rethink  how  we  design  and  build  systems”  (Warwick, 

2010). 

1.1  Research  Subject 

The  subject  of  this  research  effort  addresses  one  of  the  specific  METAII  objectives,  a 
metric  for  adaptability.  DARPA  defines  adaptability,  in  context  of  this  program,  as  the  ability  of 
a  system  to  change  easily,  quickly,  and  inexpensively  (i.e.,  with  minimum  incurrence  of  cost  and 
degradation  in  performance)  in  response  to  a  wide  spectrum  of  anticipated  and  unanticipated 
perturbation  events  exogenous  or  endogenous  to  the  system. 

1.2  Performing  Organization 

The  research  has  been  performed  by  the  Massachusetts  Institute  of  Technology  (MIT) 
Systems  Engineering  Advancement  Research  Initiative  (SEAri),  a  research  group  affiliated  with 
MIT’s  Engineering  Systems  Division  (ESD)  and  the  Center  for  Technology,  Policy  and 
Industrial  Development  (CTPID).  The  research  leveraged  MIT  SEAri’s  prior  and  ongoing 
research,  and  its  existing  infrastructure  to  support  the  effort.  The  SEAri  research  group  mission  is 
to  advance  the  theories,  methods,  and  the  effective  practice  of  systems  engineering  applied  to 
complex  socio-technical  systems  through  collaborative  research.  The  MIT  principal  investigator 
was  Dr.  Donna  H.  Rhodes,  and  the  MIT  co-principal  investigator  was  Dr.  Adam  M.  Ross.  MIT 
Professor  Daniel  E.  Hastings  was  involved  in  a  strategic  guidance  role.  A  number  of  graduate 
and  undergraduate  students  worked  on  the  project. 

1.3  Problem 

Success  in  modern  systems  is  strongly  determined  by  being  able  to  respond  to 
perturbations  on  appropriate  timescales.  Current  metrics  for  assessing  the  DARPA  adaptability 
of  a  system  suffer  from  data  accessibility  bias.  In  particular,  the  costs  for  embedding  system 
change  options,  as  well  as  executing  these  change  options,  are  much  better  characterized  than  the 
benefits  of  doing  so.  This  imbalance  is  partly  due  to  prevailing  valuation  techniques  requiring 
the  probabilities  of  outcomes  driving  the  execution  of  real  options,  as  well  as  the  specific 
“destination”  end  states  for  a  system.  The  benefit  of  more  “open-ended”  change  options  is 
therefore  undervalued  in  analysis.  Likewise,  from  an  acquisition  perspective,  the  cost  of 
embedding  options  is  typically  incurred  up-front  in  the  lifecycle  while  the  benefit  may  occur  in 
some  uncertain  future.  The  ability  to  pursue  active  value  robust  strategies  (adaptability,  or  more 
generally  changeability)  must  be  properly  valued  in  order  to  be  proactively  embedded  in  a 
system  architecture.  As  such,  addressing  the  cost-benefit  imbalance  is  essential.  Useful  metrics 
are  needed  to  inform  the  selection  of  promising  adaptable  concept  designs  for  further  analysis. 

1.4  Results 

The  research  effort  sought  to  develop  a  metric  of  adaptability,  but  what  was  discovered  is 
that  several  important  dimensions  are  in  tension  when  attempting  to  account  for  valuable 
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adaptability,  or  more  generally,  changeability.  These  dimensions  in  tension  include:  design 
versus  performance  considerations,  short  run  versus  long  run  returns,  and  context  specific  or 
general  applicability  of  the  metrics.  As  a  result  of  this  finding,  a  set  of  metrics  was  developed, 
along  with  a  valuation  approach  in  order  to  provide  guidance  in  calculation  and  use  of  these 
metrics  to  design  data  sets. 

1.4.1  Publications 

During  the  performance  of  the  contract,  one  conference  publication  was  published  and 
presented  at  the  9th  Conference  on  Systems  Engineering  Research  during  April  2011  in  Los 
Angeles,  CA.  The  paper  "A  Method  Using  Epoch-Era  Analysis  to  Identify  Valuable 
Changeability  in  System  Design,"  and  was  authored  Matthew  Fitzgerald,  Adam  Ross,  and  Donna 
Rhodes.  The  paper  is  available  in  the  proceedings  and  on  the  MIT  SEAri  website 
(http://seari.mit.edu).  The  research  team  expects  to  publish  two  additional  papers  in  2012.  In 
addition,  an  MIT  masters  thesis  related  to  the  project  will  be  forthcoming  in  May  2012  by 
graduate  student  Matthew  Fitzgerald,  and  following  publication  will  be  posted  on  the  SEAri 
website. 

1.4.2  Valuation  Approach  for  Strategic  Changeability  (VASC) 

The  synthetic  approach  to  valuating  changeability  across  the  set  of  metrics  developed  in 
this  research  consists  of  five  steps: 

Step  1 :  Set  up  data  for  epoch-era  analysis 

Step  2:  Identify  designs  of  interest 

Step  3:  Define  rule  usage  strategies 

Step  4:  Conduct  multi-epoch  changeability  analysis 

Step  5:  Conduct  era  simulation  and  analysis 

1.4.3  Adaptability  Metrics 

A  set  of  adaptability  metrics  was  developed  and  is  summarized  below: 


Aspect  of 
Valuable 
Changeability 

Acronym 

Stands  For 

Definition 

Robustness  via 
“no  change” 

NPT 

Normalized  Pareto  Trace 

%  epochs  for  which  design  is  Pareto 
efficient  in  utility/cost 

Robustness  via 
“no  change” 

fNPT 

Fuzzy  Normalized  Pareto 
Trace 

Above,  with  margin  from  Pareto 
front  allowed 

Robustness  via 
“change” 

eNPT, 

efNPT 

Effective  (Fuzzy) 
Normalized  Pareto  Trace 

Above,  considering  the  design’s  end 
state  after  transitioning 

“Value”  gap 

FPN 

Fuzzy  Pareto  Number 

%  margin  needed  to  include  design 
in  the  fuzzy  Pareto  front 

“Value”  of  a 
change 

FPS 

Fuzzy  Pareto  Shift 

Difference  in  FPN  before  and  after 
transition 
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Aspect  of 
Valuable 
Changeability 

Acronym 

Stands  For 

Definition 

“Value”  of  a 
change 

ARI 

Available  Rank  Increase 

#  of  designs  able  to  be  passed  in 
utility  via  best  possible  change 

Degree  of 
changeability 

OD 

Outdegree 

#  outgoing  transition  arcs  from  a 
design 

Degree  of 
changeability 

FOD 

Filtered  Outdegree 

Above,  considering  only  arcs  below 
a  chosen  cost  threshold 

1.4.4  Considerations  for  Deployment 

The  approach  and  set  of  metrics  developed  in  this  research  is  generally  applicable  to 
design  decision  problems  where  there  is  a  need  to  account  for  the  cost  and  benefit  of  investing  in 
and  executing  changes  in  designs.  Most  of  the  work  is  targeted  toward  analysis  and  decision 
making  during  conceptual  design,  with  low  fidelity  models  to  evaluate  alternatives,  however,  the 
approach  is  applicable  throughout  the  lifecycle  to  any  level  of  abstraction  in  the  system.  The  key 
consideration  when  applying  the  approach  later  in  the  lifecycle,  is  the  concomitant  increase  in 
computational  expense  for  evaluating  alternatives  with  higher  and  higher  fidelity  models, 
simulations,  and  tests.  In  order  to  manage  this  increase  in  evaluation  cost,  one  must  reduce  the 
breadth  of  alternatives  considered,  as  well  as  take  advantage  of  improvements  in  algorithm 
design  (to  improve  the  scaling  properties  of  the  suggested  approach)  and  parallelization  (to 
improve  raw  computing  time  for  generating  results).  The  approach  was  specifically  developed  in 
order  to  maximize  potential  for  parallelization,  as  well  as  to  focus  human-intervention  only  at  the 
beginning  and  end  of  the  process,  maximizing  the  ability  to  leverage  automation. 

1.4.5  Documented  Case  Applications 

The  research  has  resulted  in  three  documented  case  examples:  X-TOS,  Space  Tug,  and 
Satellite  Radar  System.  The  primary  purpose  of  the  application  of  the  X-TOS  case  in  this 
research  investigation  was  twofold:  (1)  to  serve  as  an  experimentation  case  to  develop  the 
metrics  assessment  approach,  and  (2)  to  test  the  various  adaptability  metrics  identified  through 
empirical  investigation.  The  primary  purpose  of  the  application  of  the  Space  Tug  case  in  this 
research  investigation  was  to  demonstrate  the  end-to-end  process  in  a  relatively  simple  case.  In 
particular,  the  application  to  the  Space  Tug  system  demonstrates  both  evaluation  of  valuable 
changeability  within  epochs  (short  run  value  of  changeability)  as  well  as  across  eras  via 
“strategies”  (long  run  value  of  changeability).  The  primary  purpose  of  the  application  of  the 
Satellite  Radar  System  case  in  this  research  investigation  was  to  demonstrate  scalability  of  the 
end-to-end  process  in  a  more  complex  case.  The  cases  are  detailed  in  Section  4  of  this  report. 


1.4.6  Conclusions 

The  research  conducted  has  developed  a  set  of  metric  for  valuating  adaptability  in  a  more 
complete  manner  than  existed  in  the  literature  to  this  point.  Some  challenges  with  scalability  to 
large  numbers  of  design  alternatives  or  long  computation  times  remain,  but  are  not  expected  to 
be  insurmountable.  A  key  benefit  that  emerged  from  the  development  of  VASC  and  its  related 
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metrics,  is  the  ability  to  not  only  quantify  the  costs  and  benefits  of  investing  in  changeability,  but 
also  a  more  holistic  ability  to  consider  the  driving  strategic  need  for  changeability,  providing 
focus  on  the  most  salient  aspects  of  the  problem.  Adaptability  is  desired  because  of  a  need  to  be 
able  to  alter  a  system  in  response  to  a  perturbation,  whether  that  perturbation  occurs  early  or  late 
in  the  lifecycle.  VASC  provides  a  means  to  consider  that  spectrum  of  perturbations  and 
empowers  analysts  with  the  tools  to  determine  and  justify  investment  in  system  changeability  to 
most  effectively  and  efficiently  overcome  those  perturbations. 
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2.  INTRODUCTION 

The  DARPA  META-II  program  goal  is  to  substantially  improve  the  design, 
manufacturing,  and  verification  of  complex  cyber-physical  systems,  and  particularly  defense  and 
aerospace  systems  such  as  ground  combat  vehicles,  airplanes,  and  rotorcraft.  The  program  is 
motivated  by  the  accelerating  complexity  of  systems.  DARPA  META  program  manager  Mr. 

Paul  Eremenko  describes  the  goal  as:  “to  rethink  how  we  design  and  build  systems”.Error! 
Bookmark  not  defined. 

The  subject  of  this  research  effort  addresses  one  of  the  specific  META-II  objectives,  a 
metric  for  adaptability.  DARPA  defines  adaptability,  in  context  of  this  program,  as  the  ability  of 
a  system  to  change  easily,  quickly,  and  inexpensively  (i.e.,  with  minimum  incurrence  of  cost  and 
degradation  in  performance)  in  response  to  a  wide  spectrum  of  anticipated  and  unanticipated 
perturbation  events  exogenous  or  endogenous  to  the  system. 

The  research  has  been  performed  by  the  MIT  SEAri,  a  research  group  affiliated  with 
MIT’s  ESD  and  the  CTPID.  The  research  leveraged  MIT  SEAri’s  prior  and  ongoing  research, 
and  its  existing  infrastructure  to  support  the  effort.  The  SEAri  research  group  mission  is  to 
advance  the  theories,  methods,  and  the  effective  practice  of  systems  engineering  applied  to 
complex  socio-technical  systems  through  collaborative  research.  The  MIT  principal  investigator 
was  Dr.  Donna  H.  Rhodes,  and  the  MIT  co-principal  investigator  was  Dr.  Adam  M.  Ross.  MIT 
Professor  Daniel  E.  Hastings  was  involved  in  a  strategic  guidance  role.  A  number  of  graduate 
and  undergraduate  students  worked  on  the  project. 

This  technical  report  provides  a  comprehensive  description  of  the  one  year  research 
effort.  Section  1  provides  an  executive  summary.  In  this  introductory  Section  2  we  describe  the 
problem,  scope  of  the  work,  background  information,  targeted  impact,  and  innovations.  In 
Section  3  we  discuss  the  methods,  assumptions,  and  procedures.  This  section  includes  the 
constructs,  methods,  and  data  sets  that  have  resulted  from  prior  research  that  were  used  as  a 
foundation  for  this  work.  Section  4  includes  the  specific  results  of  the  research,  and  a  discussion 
on  several  topics:  the  fit  with  larger  META  projects,  scalability,  incorporating  into  existing 
studies,  limitations/gaps,  and  future  research.  The  overall  conclusions  are  discussed  in  Section  5. 

Problem  Description.  Current  metrics  for  assessing  the  DARPA  adaptability1  of  a 
system  suffer  from  data  accessibility  bias.  In  particular,  the  costs  for  embedding  system  change 
options,  as  well  as  executing  these  change  options,  are  much  better  characterized  than  the 
benefits  of  doing  so.  This  imbalance  is  partly  due  to  prevailing  valuation  techniques  requiring  the 
probabilities  of  outcomes  driving  the  execution  of  real  options,  as  well  as  the  specific 
“destination”  end  states  for  a  system.  The  benefit  of  more  “open-ended”  change  options  is 
therefore  undervalued  in  analysis.  Likewise,  from  an  acquisition  perspective,  the  cost  of 
embedding  options  is  typically  incurred  up-front  in  the  lifecycle  while  the  benefit  may  occur  in 
some  uncertain  future.  The  ability  to  pursue  active  value  robust  strategies  (adaptability,  or  more 
generally  changeability)  must  be  properly  valued  in  order  to  be  proactively  embedded  in  a 
system  architecture.  As  such,  addressing  the  cost-benefit  imbalance  is  a  necessary  next  step. 


1  IMPORTANT  NOTE:  The  DARPA  term  “adaptability”  is  equivalent  to  the  MIT  SEAri  term 
“changeability’  which  encompasses  adaptability  (endogenous  change  agent)  and  flexibility  (exogenous  change 
agent).  In  this  report  we  will  use  the  term  changeability  as  synonymous  with  DARPA  adaptability. 

5 

Approved  for  public  release;  distribution  unlimited 


Scope  and  Research  Goals.  The  overall  research  objective  was  to  develop  a 
comprehensive  quantitative  metric  for  adaptability  that  can  be  traded  against  other  system 
metrics.  Previous  MIT  research  has  developed  a  rigorous  definition  and  metrics  for  changeability 
(such  as  “filtered  outdegree”  in  Ross,  2006).  This  research  has  built  on  prior  foundational  work 
by  generalizing  the  formulation  of  the  adaptability  metric,  allowing  for  independence  of 
enumeration  of  end  states  and  dependence  on  specification  of  change  mechanisms,  whose 
omission  from  previous  metrics  has  resulted  in  the  undervaluation  of  adaptability  in  systems 
architectures. 

Targeted  Impact.  Once  properly  characterized  with  appropriate  benefits,  as  well  as 
costs,  it  is  anticipated  that  more  system  acquirers  will  recognize  the  value  of  making  up-front 
investments  in  options  that  allow  for  system  changeability  at  appropriate  points  in  time  over  a 
system’s  lifecycle.  The  inclusion  of  active  value  robustness  strategies  will  result  in  both  long  run 
program  cost  savings,  as  well  as  sustainment  of  system  value  delivery  across  both  endogenous 
and  exogenous  perturbations. 

Key  Innovations.  This  research  has  several  key  innovations  that  have  not  been 
adequately  addressed  in  other  methods,  techniques,  and  research.  These  innovations  include  a 
rigorous  mathematical  generalization  of  a  “degree  of  adaptability”  metric,  identification  of 
architecture  features  that  correlate  with  adaptability,  and  creation  of  an  ability  to  more  fully 
account  for  benefits  of  adaptability,  allowing  for  justification  of  investment  in  adaptability¬ 
enabling  or  enhancing  system  features. 

The  modem  system  development  and  acquisition  environment  involves  a  number  of 
dynamic  factors  including  changing  technologies,  user  sets,  concept  of  operations,  and  threats. 
The  long  duration  and  complex  nature  of  system  development  efforts  often  results  in  the  need  to 
fix  requirements,  including  needs,  concepts,  and  technologies,  many  years  prior  to  actual  system 
operation.  A  common  approach  to  dealing  with  the  inevitable  change  facing  a  system  is  to 
encapsulate  the  future  as  fraught  with  uncertainty.  Techniques  exist  for  dealing  with  uncertainty, 
including  the  concept  of  real  options.  Early  system  architecting  and  development,  when  the 
significant  costs  and  capabilities  of  the  system  are  scoped  and  specified,  is  a  key  leverage  point 
for  making  high  impact  decisions  that  will  strongly  affect  the  ultimate  lifecycle  success  of  the 
system.  Unfortunately  analyses  to  support  this  critical  decision  are  strongly  biased  by  the 
difficulty  of  quantification  for  the  costs  and  benefits  of  embedding  flexibility  into  a  system 
architecture.  Often  the  costs  of  adaptability  are  more  easily  accessible  and  quantifiable,  due  to 
over-representation  in  the  near  term,  and  are  not  offset  by  benefits  accrued  in  the  long  term.  This 
work  has  addressed  the  need  to  more  fully  account  for  all  of  the  costs  and  benefits  on  a  common 
basis,  even  if  they  cannot  be  quantified  in  terms  of  dollars. 

Developing  a  positive  response  to  uncertainty  and  change  requires  changeability  (Fricke 
and  Schulz,  2005;  McManus  and  Hastings,  2006).  MIT  has  made  significant  progress  toward  an 
architecting  science  in  recent  years,  including  an  enriched  definition  of  and  metrics  for 
changeability.  Prior  changeability  quantification  work  tended  to  be  more  heuristic-driven 
(Fricke  et  al.,  2000;  Fricke  and  Schulz,  2005),  however,  MIT  has  introduced  theoretically- 
derived  quantifications  that  remove  the  bias  inherent  in  heuristics  (Ross  et  al.,  2008).  Current 
methods  to  quantify  ‘flexibility  and  adaptability’,  as  subtypes  of  changeability,  tend  to  be 
empirically  derived  and  therefore  are  contextually  biased  (Chen  and  Yuan,  1998;  Raj  an  et  al., 
2005;  Keese  et  al.,  2007).  This  metric  requires  a  quantification  of  whether  a  system  can  change 
itself  or  be  changed  (in  response  to  perturbations),  as  well  as  the  value  (benefit  net  cost  and  time) 
of  executing  the  change. 
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Upfront  critical  decision  regarding  whether  and  where  to  embed  adaptability  requires  not 
only  the  ability  to  quantify  adaptability,  but  also  the  ability  to  value  adaptability.  Prior  valuation 
work,  while  useful  in  certain  instances,  is  too  assumption-limited  or  is  mathematically 
inaccessible  (Nilchiani  and  Hastings,  2007),  limiting  its  usefulness  as  a  practical  method.  The 
results  of  classical  real  options  analysis  may  suffer  from  lack  of  support  from  senior  decision 
makers  due  to  esoteric  or  unreasonable  assumptions  (Shah  et  al.,  2008).  To  develop  a  decision 
maker  useful  adaptability  valuation  analysis  method,  this  research  aimed  to  develop  the  ability  to 
value  adaptability  in  a  more  rigorous  manner,  incorporating  a  theoretically  derived,  yet 
accessible,  quantification  of  adaptability. 

The  research  effort  included  three  interrelated  tasks.  The  first  was  the  quantification  of 
degrees  of  adaptability  across  the  spatial-temporal  system  architecture,  extending  beyond  the 
prior  MIT  work  on  “filtered  outdegree”  of  countable  mechanisms,  specified  end  states.  The  task 
aimed  to  develop  a  mathematical  treatment  of  quantifying  adaptability  change  paths  as  related  to 
the  number  of  mechanisms  and  number  of  end  states. 

The  second  task  was  the  identification  of  architectural  features  that  drive  an  adaptability 
metric.  This  task  developed  architectural  approaches  and  strategies  for  generating  these  types  of 
change  paths  as  a  function  of  time-space  location  within  a  system  architecture.  The  existing 
literature,  as  well  as  insights  from  recent  research  regarding  related  ilities  (such  as  flexibility  and 
survivability)  were  used  as  a  basis  to  propose  a  set  of  design  principles  and  architectural 
strategies  that  may  result  in,  or  at  least  correlate  with,  increased  adaptability  scores. 

The  third  task  focused  on  the  valuation  of  adaptability — extension  of  the  metric  to 
include  worth,  incorporating  the  value  of  pursuing  such  adaptability  by  taking  into  account  the 
effects  of  endogenous  and  exogenous  perturbations  that  may  face  the  system.  Several  techniques 
from  real  options  analysis  (Shah  et  al.,  2008)  and  Epoch-Era  Analysis  (Ross,  2006;  Ross  and 
Rhodes,  2008;  Viscito  and  Ross,  2009))  were  pursued  to  illustrate  a  metric  of  “valuably 
adaptable”,  including  what  conditions  must  hold  and  what  additional  data  would  be  required  in 
order  to  calculate  this  more  “advanced”  metric. 
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3.  METHODS,  ASSUMPTIONS  and  PROCEDURES 


3.1  Methods  and  Assumptions 

Prior  MIT  research  on  adaptability  (changeability)  has  developed  a  theoretically  sound, 
context  bias-free  definition  of  system  change,  as  well  as  a  taxonomy  of  types  of  change.  This 
definition  and  taxonomy  provide  a  basis  for  specifying,  designing,  and  verifying  that  systems  are 
capable  of  being  changeable  (Ross  et  al.,  2008).  This  prior  work,  highlighted  in  the  subsections 
that  follow,  provided  foundational  constructs  and  methods  for  further  development  in  this 
research. 

Assumption.  The  DARPA  META-II  program  defines  adaptability  as  “the  ability  of  a 
system  to  change  easily,  quickly,  and  inexpensively  (i.e.,  with  minimum  incurrence  of  cost  and 
degradation  in  performance)  in  response  to  a  wide  spectrum  of  anticipated  and  unanticipated 
perturbation  events  exogenous  or  endogenous  to  the  system.  The  explicit  strategy  in  this, 
equivalent  to  maximizing  utility,  is  only  one  of  a  number  of  possible  strategies  that  we  assume 
are  of  interest  to  DARPA  in  spite  of  not  being  explicitly  stated.  In  this  work  we  also  investigated 
strategies  for  survivability  of  the  system,  maximizing  efficiency,  and  maximizing  profit. 

3.1.1  Change  Events  as  Paths 

The  construct  of  “change  events  as  paths”  is  fundamental  for  a  more  precise 
understanding  of  a  system  change.  A  system  change  event  can  be  characterized  with  three 
elements:  (1)  the  agent  of  change,  (2)  the  mechanism  of  change,  and  (3)  the  effect  of  change,  as 
illustrated  in  Figure  1.  The  agent  of  change  is  the  instigator,  or  force,  for  the  change.  The  role  of 
change  agent  can  be  intentional  or  implied,  but  always  requires  the  ability  to  set  a  change  in 
motion.  The  mechanism  of  change  describes  the  path  taken  in  order  to  reach  a  future  state  from 
the  present  state,  including  any  costs,  both  time  and  money,  incurred.  Examples  of  mechanisms 
include  the  execution  of  real  options,  such  as  the  swapping  of  modular  components,  or  the 
procurement  of  additional  system  elements.  The  effect  of  change  is  the  actual  difference  between 
the  origin  and  destination  states. 


Aqent 

State  1  State  2 

, — Aqent  type  .nH  “Cost r  incurred  ./'S 

^TA  Mechanism  TA’ 

Figure  1.  Change  Defined  as  State  Transition 


The  change  described  in  Figure  1  is  a  simple  case  of  one  particular  change.  In  the  agent- 
mechanism-effect  representation,  a  particular  change  is  represented  by  a  path.  The  changeability 
of  a  system  is  determined  by  how  easily  it  can  undergo  various  changes.  Figure  2  shows  an 
example  of  an  expanded  view  with  multiple  change  paths  enumerated,  where  a  change  agent 
external  to  the  system  is  termed  a  flexible  change,  and  where  the  change  agent  internal  to  the 
system  is  an  ‘adaptable’  change. 
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Figure  2.  System  Changes  Depicted  using  Agents,  Mechanisms,  and  Effects 


Table  1  enumerates  the  example  agents,  mechanisms,  and  effects  shown  in  Figure  2.  For 
a  particular  system,  many  agents,  mechanisms,  and  end  states  may  be  possible. 


Tal 

ble  1.  Agents,  Mechanisms,  Effects,  and  Pal 

;hs  Shown  in  Figure  2 

Element 

Description 

As  Illustrated  in  Figure  2 

Change  Agent 

The  force  instigator  for  the  change  to  occur,  for 
example  humans,  software,  Mother  Nature,  etc. 

a,  P 

Change  Mechanism 

The  particular  path  the  system  must  take  in  order 
to  transition  from  its  prior  to  its  post  state, 
including  conditions,  resources,  and  constraints 

1,2 

Change  Effect 

The  difference  in  states  before  and  after  a  change 
has  taken  place. 

A’-A,  B’-A,  C’-A 

Potential  Paths 

The  potential  paths  for  the  system  to  change  from 
one  state  to  another. 

a:A-l-A’,  a:A-l-B’ 

P:A-2-A\  P:A-2-C’ 

Figure  3  illustrates  the  notion  of  “perturbation”  as  an  instigator  for  a  change  pathway, 
perturbation-agent-mechanism-effect.  A  perturbation  can  be  exogenous  to  (outside  of)  or 
endogenous  to  (inside  of)  the  system.  The  decision  is  made  as  to  whether  to  execute  a  pathway. 


Figure  3.  Change  Pathway  with  Perturbation-Agent-Mechanism-State  Change 
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3.1.2  Changeability  Taxonomy  Based  on  Change  Pathway 

The  change  agent  location  (internal  or  external  to  system)  is  a  useful  taxonomic 
distinction  for  classifying  change.  If  the  change  agent  is  external  to  the  system,  then  the  change 
under  consideration  is  a  flexible-type  change  in  this  taxonomy.  If  the  change  agent  is  internal  to 
the  system,  then  the  change  under  consideration  is  an  adaptable- type  change.  Note  that 
depending  on  the  particular  change  being  considered,  a  single  system  can  be  both  flexible  and 
adaptable.  The  definition  of  the  system  boundary  must  be  explicitly  defined  in  order  to  remove 
ambiguity  when  discussing  whether  a  change  should  be  considered  as  flexible  or  adaptable 

3.1.3  Epoch-Era  Analysis 

Quantifying  the  changeability  of  a  system  necessitates  bringing  the  dynamic  aspects  into 
consideration,  as  well  as  a  temporal  value-based  perspective.  Epoch-Era  Analysis  (EEA)  (Ross, 
2006;  Ross  and  Rhodes,  2008)  provides  an  approach  for  conceptualizing  system  timelines  using 
natural  value  centric  timescales,  wherein  the  context  itself  defines  the  timescales.  The  full 
lifespan  of  a  system  is  referred  to  as  the  System  Era,  which  can  be  decomposed  into  Epochs.  An 
Epoch  is  a  period  for  which  the  system  context  has  constant  value  expectations.  Each  fixed 
context  is  characterized  by  static  constraints,  available  design  concepts,  available  technology, 
and  articulated  attributes.  As  exogenous  changes  (e.g.,  new  threat,  availability  of  a  new 
technology,  new  policy,  etc.)  trigger  the  start  of  a  new  epoch,  the  system  may  need  to  transform 
in  order  to  sustain  value  in  the  new  context,  or  else  it  may  fail  to  meet  expectations  as  defined  for 
this  new  context,  as  illustrated  in  Figure  4  below. 


Needs  ( performance r  expectations) 


Expectation  2 

Context 

2 

Expectation  1 

Context 

2 

Context 

1 

I  Changed 
System 


IUndtanged 
System 


Context  Context 
3  4 


Time 

(epochs) 


Epoch  1  Epoch  2  Epoch  3  Epoch  4  Epoch  5 
- Y~ — ' 

Short  run 


Long  run  - 


Legend 
•  System 

—  ►  System  Trajectory 
Expectations 


Figure  4.  System  Needs  versus  Expectations  across  Epochs  of  the  System  Era 


Figure  4  illustrates  the  temporal  evolution  of  a  system  as  needs  and  contexts  change.  A 
system  exists  in  Context  1  in  Epoch  1  and  has  performance  exceeding  expectations. 

Expectations  are  represented  by  a  band  (gray  shaded  in  figure)  capturing  the  range  from 
minimally  acceptable  to  the  highest  of  expectations.  In  Epoch  2,  the  context  changes  to  Context 
2  and  the  performance  is  degraded.  Expectations  are  still  met  with  the  same  system,  so  the 
system  is  relatively  robust  to  the  change  in  context.  A  change  in  expectation  is  shown  in  Epoch 
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3,  with  the  context  (“Context  2”)  remaining  the  same  as  the  second  epoch;  now  the  still 
unchanged  system  exhibits  value  robustness  since  it  maintains  value  delivery  in  spite  of  changes 
in  expectations.  In  Epoch  4,  the  system  shows  versatility  by  continuing  to  satisfy  expectations 
despite  the  introduction  of  a  new  metric  of  need.  Notice  that  even  though  the  system  no  longer 
exceeds  all  expectations,  it  still  does  exceed  the  minimally  acceptable  level  and  thus  is  still 
successful.  Finally,  in  Epoch  5,  a  change  in  context  and  a  boost  in  expectations  are  too  much  for 
the  system  as-is;  in  this  case  the  system  must  change  in  order  to  remain  successful.  If  the  system 
is  capable  of  changing  at  acceptable  cost,  it  is  deemed  flexible  or  adaptable,  depending  on  the 
type  of  change  desired  (as  determined  by  change  agent  -  flexible,  external  agent;  adaptable, 
internal  change  agent). 

Epoch-Era  Analysis  provides  an  approach  for  visualization  and  a  structured  way  to  think 
about  the  temporal  system  value  environment,  and  is  used  in  our  research  approach  as  a  useful 
mechanism  for  specifying  strategies  over  time.  The  Epoch  timelines  can  be  assessed  at  any  point 
during  the  system  lifecycle,  not  only  during  early  conceptual  design.  For  analysis  purposes, 
epochs  can  be  known  in  advance,  or  in  the  moment,  and  can  be  deterministic,  or  probabilistic.  As 
such,  mathematical  treatment  of  the  paths,  costs,  utilities,  and  times  must  appropriately  match  the 
uncertainty  level  of  the  data. 

Selection  of  the  system  Epoch  end  state  is  dependent  on  the  strategy  for  the  Epoch.  No 
absolute  correct  or  “best”  design  exists  without  subjectively  specifying  the  “best”  strategy. 
Strategies  can  include  seeking  maximum  utility,  minimum  cost,  minimum  time,  minimum  risk, 
or  any  combination,  among  others.  Strategies  themselves  can  be  predictive,  adaptive,  or  static, 
meaning  an  analyst  can  use  them  to  predict  “best”  paths  for  a  system  given  present  knowledge  of 
the  future,  to  adapt  given  new  information  about  current  and  future  epochs,  or  to  statically  drive 
a  particular  agenda  for  a  fixed  set  of  objectives  and  technology  in  a  changing  world.  The  system 
analyst  can  use  the  epoch  analysis  while  the  system  is  in  operation,  continuously  updating 
probabilities  and  value  data  to  determine  the  “best”  path  to  other  designs,  as  well  as  the  “best” 
goal  design  to  pursue  in  each  epoch. 

3.1.4  Tradespace  Networks 

The  typical  tradespace  plot  displays  the  system  designs  on  a  Cost-Utility  space,  showing 
the  resources  required  (cost)  and  benefit  delivered  (utility)  for  systems  in  a  concise  format.  A 
Pareto  Set  characterizes  those  “non-dominated”  designs  of  highest  utility  at  a  given  cost,  across 
all  costs,  or  those  of  lowest  cost  at  a  given  utility,  across  all  utilities.  This  set  often  shows  the 
tradeoff  of  cost  incurred  for  increased  value.  Considering  each  design  as  a  potential  starting  or 
ending  state  for  change,  the  tradespace  frame  suggests  a  mechanism  for  considering  the 
changeability  of  system  designs.  If  in  addition  to  specifying  design  parameters  (static 
representations  of  a  system)  designers  also  specify  transition  paths  (dynamic  change 
opportunities),  a  traditional  tradespace  can  become  a  tradespace  network  (Ross  and  Hastings, 
2006). 

A  network  is  a  model  representation  of  nodes  and  arcs.  Each  node  represents  a  location, 
or  state,  with  each  arc  representing  a  path  that  connects  particular  nodes.  In  a  tradespace 
network,  system  designs  are  nodes  and  the  transition  paths  are  arcs.  Each  arc  represents  a 
transition  with  a  “cost”  in  terms  of  both  dollars  and  time.  The  transition  paths  represent  each  of 
the  potential  change  mechanisms,  with  change  agent,  available  to  a  particular  design.  Figure  5 
shows  a  traditional  static  utility-cost  tradespace  transformed  into  a  tradespace  network  after  the 
specification  of  transition  rules,  which  are  used  to  generate  transition  paths  between  design 
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nodes.  Designs  that  can  follow  more  transition  paths  will  have  more  outgoing  arcs  connecting  it 
to  other  designs. 


Figure  5.  Tradespace  Transformed  into  a  Tradespace  Network  through  Transition 

Rules 


Figure  6  illustrates  that  when  counting  paths,  one  includes  the  agent-mechanism 
combination  as  a  unique  path.  Each  path  has  an  associated  cost  with  it,  given  the  mechanism  that 
is  used  to  move  the  system  from  one  state  to  another  state 


3.1.5  Filtered  Outdegree 

The  tradespace  network  representation  provides  a  foundation  for  measuring  changeability 
paths,  given  that  each  path  will  have  a  “cost”  associated  with  its  execution.  Each  decision  maker 
will  have  an  acceptability  threshold  for  time  or  money  spent  for  enacting  change.  The  number  of 
outgoing  arcs  from  a  particular  design  is  called  the  outdegree  for  that  design,  as  illustrated  in 
Figure  8  (left).  The  number  of  outgoing  arcs  from  a  particular  design  whose  cost  is  less  than  the 
acceptability  threshold,  C,  is  the  filtered  outdegree  for  that  design,  as  illustrated  in  Figure  7 
(right)  (Ross,  2006).  The  fdtered  outdegree  is  a  quantification  of  the  apparent  changeability  for  a 
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design  for  a  decision  maker.  The  higher  the  filtered  outdegree  of  a  design,  the  more  changeable  it 
is  to  that  decision  maker.  In  the  figure  below,  the  outdegree  counts  the  total  number  of  change 
paths  from  a  given  design,  state  1 :  A,  to  future  designs,  states  2:  A’,  B’,  and  C’,  shown  with 
outdegree  of  four;  (right)  The  filtered  outdegree  counts  the  number  of  change  paths  with 
acceptable  cost,  from  a  given  design,  state  1 :  A,  to  future  designs,  states  2:  A’,  shown  with 
filtered  outdegree  of  two. 


The  objective,  coupled  with  subjective  nature,  of  the  filtered  outdegree  captures  the 
apparent  relativity  in  perceived  changeability  of  various  designs:  what  may  be  changeable  to  one 
decision  maker  may  not  be  perceived  changeable  to  another.  The  subjective  acceptability 
threshold  differentiates  the  results  per  decision  maker.  Acceptability  threshold  “cost”  can  be  on 
dollars,  time,  or  any  other  resource  that  must  be  “spent”  in  order  to  follow  a  path.  The  objective 
outdegree  calculation  provides  a  mechanism  for  system  designers  to  explicitly  improve  the 
potential  changeability  of  a  system:  increase  the  number  of  outgoing  arcs  (add  new  transition 
rules),  or  reduce  the  cost  of  following  outgoing  arcs  (increase  the  likelihood  for  arcs  to  cost  less 
than  acceptability  threshold).  The  subjectivity  in  the  filtered  outdegree  means  that  the  setting  of 
the  threshold  is  subjective  to  the  particular  decision  maker  and  his  preferences  for  spending 
resources  for  change.  The  full  outdegree,  without  filter,  is  an  objective  quantity  upon  which  all 
people  will  agree,  given  a  set  of  enumerated  design  variables  and  a  set  of  transition  rules. 

3.1.6  Real  Options  Analysis  for  Valuing  Changeability 

Quantifying  changeability  is  different  than  valuing  changeability.  The  former  is  about 
“whether  something  is  changeable,”  while  the  latter  is  about  “what  is  it  worth”  to  have 
something  changeable.  Foundational  work  has  been  done  in  regard  to  valuation  of  real  options 
and  additional  research  is  ongoing.  The  field  of  real  options  was  motivated  by  the  desire  to  apply 
quantitative  financial  options  valuation  methods  to  capital  investment  decisions.  The  term  “real 
options”  was  first  used  by  (Myers,  1984)  in  the  context  of  strategic  decision  making,  where 
“real”  refers  to  the  fact  that  the  underlying  asset  is  real  rather  than  financial.  A  “real  option” 
gives  the  decision  maker  the  right,  but  not  the  obligation,  to  exercise  an  action  or  decision  at  a 
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later  point  in  time.  The  concept  of  real  options  analysis  is  to  value  investment  decisions  by 
taking  into  account  the  options  that  are  available  to  the  decision  maker  in  the  future.  In  a  general 
sense,  any  action  or  decision  that  can  be  taken  in  the  future  can  be  considered  to  be  a  real  option. 

Prior  work  on  valuing  flexibility  has  used  real  options  analysis  (Trigeorgis,  1998).  To 
effectively  use  the  real  options  approach,  quantitative  representation  of  the  option’s  cost,  future 
uncertainty,  and  possible  exercised  option  outcomes  are  needed.  Valuation  of  real  options  is 
done  through  mathematically  collapsing  future  uncertain  costs  and  benefits  of  the  option  into  a 
common  present  value.  Real  options  valuation  has  traditionally  been  applied  to  valuing  business 
investment  decisions  under  uncertainty  (Copeland  and  Antikarov,  1998),  but  more  recently  have 
been  applied  to  value  flexibility  in  the  context  of  system  design  under  uncertainty(de  Neufville, 
2003;  de  Week  et  al.,  2004;  Wang  and  de  Neufville,  2006). 

Three  major  approaches  to  valuing  financial  options  are:  Black-Scholes,  binomial 
pricing,  and  simulation  (Black  and  Scholes,  1973;  Cox  et  al.,  1979;  Boyle  1977).  All  of  these 
models  have  been  used  to  value  real  options.  However,  the  assumptions  underlying  the  Black- 
Scholes  model  do  not  translate  well  to  real  options.  The  binomial  pricing  model  and  Monte  Carlo 
simulation  have  been  popular  in  valuing  real  options.  Besides  financial  valuation  models, 
decision  analysis  has  been  used  to  value  real  options.  Decision  analysis  involves  constructing  a 
tree  where  the  layers  of  nodes  represent  decision  and  chance  outcomes  alternatively. 
Uncertainties  are  modeled  with  probabilities  of  chance  nodes.  Decision  analysis  calculates  the 
best  decisions  by  maximizing  the  expected  value  of  the  outcomes.  Real  options  valuation 
methods  provide  a  means  for  quantitatively  assessing  decisions  under  uncertainty,  but  additional 
research  is  necessary  to  validate  and  evolve  these  methods,  as  real  options  differ  significantly 
from  financial  options.  For  example,  real  options  often  have  a  high  carrying  cost,  not  typically 
incurred  by  financial  options.  Additionally,  the  execution  of  a  real  option  may  impact  one’s 
ability  to  exercise  other  real  options.  This  coupling  between  real  options  introduces  an 
additional  cost  for  consideration  during  the  analysis,  which  is  not  addressed  in  classical  financial 
real  options  analytic  methods. 

A  distinction  has  been  drawn  between  1)  real  options  "on"  projects  (Copeland  and 
Antikarov,  2001),  referring  to  strategic  decisions  regarding  project  investments;  and  2)  real 
options  "in"  projects  (Wang  and  de  Neufville,  2006),  which  refers  to  engineering  design 
decisions.  The  relationship  between  real  options  "on"  and  "in"  projects  is  the  subject  of  research 
in  the  MIT  SEAri  group  (Mikaelian,  2008)  examining  under  what  situations  it  would  make  sense 
to  invest  in  real  options  "in"  versus  "on"  projects.  For  example,  given  the  uncertain  space  system 
acquisition  environment,  how  decisions  can  be  made  regarding  whether  to  invest  in 
changeability  of  a  given  spacecraft  design  versus  investment  in  different  missions  or 
technologies. 

3.2  Procedures 

The  research  technical  approach  involved  three  major  research  tasks.  The  first  was  to 
develop  an  overall  metrics  assessment  approach  for  guiding  the  means  to  select  appropriate 
metrics  for  investigation  and  to  evaluate  these  metrics  in  an  appropriate  time-dependent  manner. 
The  second  task  was  to  develop  supporting  metrics  including  the  degree  of  changeability  and 
value  of  changeability,  as  well  as  determining  the  applicability  of  each  metric  to  the  appropriate 
temporal  strategy  and  data  availability.  The  third  task  was  to  apply  each  metric  to  multiple  data 
sets  to  evaluate  candidate  metrics,  to  demonstrate  end-to-end  application  of  the  approach,  and  to 
demonstrate  scalability. 
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In  addition,  the  research  team  canvassed  empirical  data  to  identify  a  list  of  change 
mechanisms  employed  on  real  cyber-physical  systems  that  can  assist  system  designers  in 
proposing  change  mechanisms  for  consideration. 

Software  was  developed  by  the  research  team  to  demonstrate  the  end-to-end  approach. 

3.2.1  Assumption:  Characteristics  of  Data  Sets 

The  following  are  assumptions  regarding  the  existing  characteristics  of  data  sets  used  in 
analysis: 

•  Data  to  characterize  design  alternatives  (attributes,  design  variables)  (Figure  8) 

•  Clearly  defined  context  variables  affecting  perceived  system  value 

•  Variables  needed  to  differentiate  epochs  (Figure  9) 

•  Must  affect  value  in  a  significant/meaningful  way  for  useful  information  to  exist 
beyond  a  single  epoch 

•  Change  mechanism  and  cost  data 

•  Requisite  information  for  chosen  changeability  value  metric 

•  Value  must  be  able  to  be  calculated  for  each  epoch 
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4.  RESULTS  AND  DISCUSSION 


4.1  Results 

4.1.1  Framing  Concepts  Leveraged  from  Prior  Work 

4.1.1. 1  Change  “Rules”  and  Mechanisms 

To  clarify  the  concept  of  change  mechanisms  and  transition  rules,  the  following  describes 
these  two  concepts.  A  change  mechanism  is  a  method  by  which  the  system  is  change.  For 
example  “bum  on  board  fuel”  change  mechanism  results  in  a  change  in  satellite  orbit,  costing 
“extra  ops  cost”  for  executing  the  maneuver  (in  this  case  the  system  “state”  includes  the 
operating  orbit).  A  transition  rule,  also  called  change  rule,  is  an  algorithm  that  determines 
whether  two  proposed  “states”  are  connected  through  a  particular  change  mechanism.  For 
example:  “compare  two  ‘states’  and  if  difference  is  only  fuel  and  orbit  location,  then  if  fuel 
difference  is  equal  to  amount  burned  to  achieve  orbit  difference,  then  states  have  directed 
accessibility  via  change  mechanism  for  cost  determined  by  that  mechanism.”  The  change  rule  is 
an  operationalization  of  the  concept  of  change  mechanism  in  order  to  allow  for  computationally 
generated  and  evaluated  alternative  “paths”  in  a  tradespace  network,  greatly  automating  the 
analysis  process. 

4.1. 1.2  “Degree  of’  Change 

One  of  the  essential  concepts  to  address  in  the  research  regarded  the  number  of  possible 
system  end  states  reachable  through  available  change  mechanisms.  In  some  sense,  if  a  design  has 
more  reachable  end  states,  that  design  is  more  changeable.  Figure  10  displays  a  four  quadrant 
view  of  differing  numbers  of  mechanisms  and  end  states.  A  given  change  mechanism  may  have 
some  number  of  countable  or  uncountable  end  states.  For  example,  a  sleep  number  bed  has  100 
“levels”  for  firmness,  corresponding  to  99  alternative  end  states  from  a  given  starting  state  using 
the  change  mechanism  of  dialing  the  controller  to  inflate  or  deflate  mattress-embedded  air 
bladders  using  a  pump.  Alternatively,  a  design  might  have  available  more  than  one  change 
mechanism,  which  also  adds  to  the  number  of  potential  end  states. 


#  mechanisms 


Figure  10.  Degree  of  Changeability  as  a  Function  of  Number  of  Mechanisms  and 

End  States 
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Prior  research  focused  in  the  lower  left  quadrant  of  Figure  10,  and  a  goal  of  this  research 
effort  was  to  expand  to  the  other  quadrants.  On  investigating  the  heuristic  that  “more  end  states 
is  better”  resulted  in  the  following  insights  for  why  the  number  paths  to  end  states  might  matter. 

1.  Mechanism  blocking.  One  can  conceive  of  instances  where  a  particular  change 
mechanism  might  become  blocked,  that  is,  unable  to  be  executed.  This  blocking 
could  occur  as  a  result  of  a  system  failure,  or  imposed  constraint,  such  as  policy.  For 
example  a  change  mechanism  may  require  the  execution  of  prior-arranged  contract 
agreement,  but  an  ensuing  policy  directive  prevents  such  relationship  from  taking 
place.  Or  one  might  purchase  spare  parts  to  allow  for  a  “repair”  change  mechanism, 
only  to  find  that  the  expert  knowledge  for  conducting  the  “repair”  was  not  available 
when  needed  years  later.  Having  more  change  mechanisms  allows  a  design  to  retain 
changeability  even  when  one  or  more  change  mechanisms  are  blocked. 

2.  Mechanism  paring.  One  can  also  conceive  of  instances  where  particular  end  states 
may  no  longer  be  available  within  execution  of  a  particular  change  mechanism.  For 
example,  the  pump  needed  in  the  sleep  number  bed  example  from  above  might 
become  mechanically  degraded  resulting  in  a  smaller  range  of  possible  firmness 
levels  from  100  to  30.  Certain  destination  orbits  for  a  deployed  satellite  might  become 
unusable  due  to  orbital  debris  accumulating  later  in  the  lifecycle  (thereby  reducing  the 
number  of  end  states  for  the  “change  orbit”  change  mechanism.  Having  more  possible 
end  states  for  a  given  change  mechanism  allows  a  design  to  retain  some  changeability 
even  when  one  or  more  end  states  are  pared  from  a  change  mechanism. 

3.  Uncertainty  in  desired  goal  end  state.  One  may  also  recognize  that  the  desirability  of 
a  particular  end  state  may  be  context  dependent.  That  is,  the  particular  mission  in 
operations  may  change  over  time  and  the  target  end  state  for  a  system  may  not  be 
what  was  anticipated  earlier  in  the  lifecycle.  Having  more  possible  end  states  allows 
for  a  design  to  have  a  higher  likelihood  of  having  a  “good”  end  state  available  when 
the  definition  of  “good”  changes  over  time. 

It  is  for  the  three  reasons  above,  and  shown  in  Figure  11,  that  the  number  of  paths,  which 
result  from  the  available  change  mechanisms  as  well  as  possible  end  states,  is  related  to  the 
concept  of  valuable  changeability  and  must  be  incorporated  into  the  metrics. 


Figure  11.  Justification  for  Need  to  Account  for  Number  of  Change  Paths 
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It  was  determined  that  the  number  of  paths  is  meaningful  for  accounting  for 
changeability,  however  the  enumeration  of  possible  end  states  may  become  an  intractable 
problem  for  change  mechanisms  with  uncountable  number  of  end  states.  Upon  further 
investigation,  the  research  team  decided  to  recognize  that  there  may  be  value  in  uncountable 
number  of  end  states,  however,  value  is  determined  only  when  change  mechanisms  are  actually 
executed  to  a  particular  end  state.  This  means  the  value  of  the  path  is  dependent  on  the  value  of 
the  “best”  end  state. 

In  order  to  clarify  the  tension  between  number  (i.e.  “degree”)  of  change  paths,  and  the 
value  (i.e.  “magnitude”)  of  the  end  states  explicitly  in  the  changeability  metrics,  the  concept  of 
rule  execution  strategy,  or  just  strategy,  was  proposed.  The  concept  of  strategy  encapsulates  the 
idea  that  “value  is  derived  from  changeability  only  with  executed  changes.”  A  strategy  is  a 
statement  of  how  and  when  a  stakeholder  plans  to  execute  any  changeability  options  in  the 
system.  For  example,  “maximize  utility,”  or  “exercise  for  survival  only.”  Given  a  defined 
tradespace  network  and  epoch,  a  strategy  will  select  the  “best”  transition  (if  any)  that  should  be 
utilized  from  each  design  point,  as  can  be  seen  in  Figure  12. 


Strategy:  maximize  utility 


Strategy:  maximize  utility 
without  increasing  costs 


Figure  12.  “Best”  Path  Selection  Determined  by  Rule  Execution  Strategy 


The  concept  of  strategy  reduces  the  burden  of  enumerating  all  possible  end  states.  In 
practice,  only  “good  enough”  end  states  need  to  enumerated  in  order  for  a  strategy  to  show  value 
in  the  changeability.  Enumeration  of  better  end  states  will  result  in  higher  value  for  the 
changeability  given  a  strategy.  In  this  way,  confidence  scales  with  effort  and  results  can  be 
gained  without  having  to  spend  exhaustive  effort  to  enumerate  end  states.  In  fact,  numerical 
optimization  and  search  methods  can  be  used  to  generate  target  end  states  for  a  given  strategy 
without  having  to  enumerate  full  tradespace  networks.  This  numerical  search  approach  is 
recommended  for  future  research. 
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4.1. 1.3  Addressing  “Counting”  and  “Magnitude”  Value 

Using  the  strategy-driven  approach  described  in  the  last  section,  one  can  apply  change 
rules  to  a  tradespace  to  generate  tradespace  networks  for  each  change  mechanism,  and  then 
simplify  the  tradespace  network  by  selecting  the  “best”  path  for  each  considered  strategy. 

As  can  be  seen  in  Figure  13,  the  application  of  one  (or  more)  strategies  can  vastly 
simplify  a  tradespace  network,  which  represents  the  most  valuable  change  path  (change 
execution)  for  a  given  design-epoch  pair.  This  then  captures  the  “magnitude”  aspect  of  valuable 
changeability.  In  order  to  re-incorporate  the  “counting”  value  of  changeability,  one  must  look 
across  many  epochs,  both  time-independent,  and  time-ordered,  to  see  why  having  more  than  one 
of  these  valuable  paths  is  worthwhile.  This  should  account  for  blocking,  paring,  and  uncertainty 
of  desired  end  states.  Having  more  end  states  will  have  an  impact  on:  being  more  likely  to  have  a 
high  value  transition  under  a  given  strategy;  more  likely  to  be  valuable  across  multiple 
alternative  strategies,  more  likely  to  retain  valuable  changeability  when  subject  to  unforeseen 
disturbances,  such  as  loss  of  change  mechanisms. 


Full  Tradespace  Network  Simplified  Tradespace  Network 


Cost  Cost 

Figure  13.  Tradespace  Network  Simplification  via  Strategy 


4.1. 1.4  Using  Strategies  for  Short  Run  and  Long  Run  Valuations 

In  addition  to  simplifying  the  tradespace  network,  the  application  of  strategies  helps  to 
explicitly  communicate  various  “usage”  approaches  to  how  changeability  could  be  exploited, 
both  in  the  short  term  (across  a  single  epoch  shift)  and  in  the  long  term  (across  an  era). 

Strategies,  such  as  “maximize  utility”  or  “minimize  operations  costs”  determine  when  to 
execute  a  change  mechanism,  typically  in  response  to  a  perturbation,  such  as  an  epoch  shift  or 
disturbance.  Across  an  era,  many  such  perturbations  might  occur.  The  strategy  is  defined  across 
an  era  and  could  be  homogenous  (applied  in  the  same  manner  across  any  epochs)  or 
heterogeneous  (applied  differently  across  particular  epochs),  as  seen  in  Figure  14.  Strategy 
formulation  itself  is  not  addressed  in  this  research,  however,  this  research  does  allow  for 
comparison  of  the  outcome  of  different  strategies  and  could  be  used  to  explicitly  communicate 
the  difference  in  value  of  a  change  mechanism  in  the  short  run  or  in  the  long  run. 
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Figure  14.  Epoch  and  Era  Strategies  Determine  Value  of  Changeability 


Looking  across  an  era,  automated  analysis  can  be  performed  in  order  to  compare  the 
likelihood  of  executing  particular  change  mechanisms  as  a  function  of  strategy,  as  seen 
notionally  in  Figure  15.  In  this  way,  designers,  analysts,  and  decision  makers  can  see  the 
implications  of  strategy  on  design  choices  to  enable  changeability.  In  particular,  if  certain  change 
mechanisms  are  more  feasible  than  others,  the  feasibility  may  constrain  which  strategies  will 
extract  the  most  value.  This  type  of  strategy  comparison  analysis  will  be  conducted  in  the  case 
studies  in  later  sections  of  the  report. 


Likelihood  of  executing  for  strategy 


^^~~~^St  rategy 
Rule 

Maximize 

Value 

Ma  intain 

Value 

Survive 

Overall  Rule 

Attractiveness 

Rl:  Plane  Change 

g 

5% 

G 

0% 

G 

0% 

2% 

R2:  Apogee  Burn 

20% 

G 

10% 

© 

40% 

,1  23% 

R3:  Perigee  Burn 

o 

5% 

G 

4% 

20% 

D 

10% 

R4:  Plane  Tug 

© 

0% 

G 

0% 

G 

0% 

.gill  o% 

R5:  Apogee  Tug 

20% 

0% 

G 

0% 

ai|||j  7% 

R6:  Perigee  Tug 

10% 

G 

0% 

G 

0% 

Jfl  3% 

R7:  Space  Refuel 

o 

5% 

O 

0% 

G 

0% 

2% 

R8:  Add  Sat 

o 

30% 

O 

1% 

G 

5% 

1 

12% 

Do  nothing 

5% 

© 

85% 

© 

35% 

.III 

42% 

Figure  15.  Attractiveness  of  Change  Rules  by  Strategy  across  an  Era 


4.1.2  Development  of  Metrics 

A  primary  goal  of  this  research  effort  is  to  develop  a  metric  of  “adaptability”  (herein 
referenced  as  “changeability).  In  order  to  develop  the  metric,  a  canvassing  of  existing  metrics 
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was  conducted,  as  well  as  a  description  of  characteristics  of  a  good  metric,  both  of  which  are 
described  below. 

4.1.2. 1  Necessity  of  Multiple  Metrics  for  Changeability 

Our  investigation  revealed  that  the  multi-aspect  nature  of  “valuable  changeability” 
requires  more  than  one  metric.  A  set  of  metrics  is  needed  to  evaluate  valuable  changeability, 
determined  by  the  system  value  sustainment  strategy  and  context,  as  well  as  the  availability 
of  data.  Metrics  for  valuable  changeability  must  address  the  following  tradeoffs: 

-  Design  vs.  Value 

•  Function  of  design-only 

•  Function  of  value-only 

-  Short  run  vs.  long  run 

•  Per  epoch  evaluated 

•  Across  era  evaluated 

-  Context  specific  vs.  general 

•  Epoch-specific 

•  Epoch-general 

These  dimensions  are  illustrated  in  Figure  16,  with  example  metrics  that  have  been  used 
in  the  literature  that  represent  one  of  the  aspects. 


f(design-only)  f(value-only)  f(per  epoch)  f(perera)  f(epochj),  i=M  fifepochj),  Vi 


Examples: 

Number  of 
spares 


Excess 

capacity 


Number  of 
options  during 
fabrication 


Number  of  Sensitivity  to  Sensitivity 

options  across  specific  available  to  changes 
lifecycle  technology  in  budget 


Figure  16.  Conflicting  Dimensions  of  Aspects  to  Account  for  Valuable  Changeability 


The  metrics  assessment  approach  guides  the  selection  of  appropriate  metrics  for  a  given  valuable 
changeability  assessment. 

4.1.2.2  Empirical  Metrics  Comparison 

Our  early  investigation  included  identifying  metrics  in  the  literature  and  in  our  prior  work 
that  would  be  candidates  for  evaluation.  We  qualitatively  assessed  these  metrics,  and  identified 
the  pros  and  cons,  as  shown  in  Table  2. 
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Table  2.  Comparison  of  Metrics  Studied  in  Investigation 


Filtered 

Outdegree 


FOD'(c)  (c)] 


Counts  #  of  cost- 
acceptable  change 
paths 


Conceptually 
simple,  mostly 
function  of  design- 
only 


Does  not  take  into 
account  benefit  of 
end  states,  or 
degrees  of  cost 


Normalized 

Filtered 

Outdegree 


Non-unique  fraction  of  Gives  sense  of 

tradespace  accessible  "accessibility"  of 

design  space 


(Same  as  FOD) 
Multiple  paths  to 
same  point  inflate 
measure 


VWFO  (vl) 


VWFO^(c)=  — i-^[sgnf  um1  -  u) 


n*Arc*(c)l 


Fraction  of  next” 
epoch  utility-changing 
cost-acceptable  paths 
available  “now" 


Accounts  some  for 
epoch-dependent 
destination  benefit 


Gains  and  losses  can 
cancel  requires 
ordered  pair-wise 
epoch  analysis 


VWFO  (v2) 


Fraction  of  tradespace  Epoch-unique 

with  utility-enhancing  measure 
cost-acceptable 
change  paths 


Does  not  take  into 
account  degrees  of 
cost  or  benefit; 


Avail.  Rank 
Increase 


AKI”{c 

/)  1 

Counts  “best"  rank 
improvement  of 
change  per  rule 


Avoids  end  state 

enumeration 

problems 


Rank  loses  "degree 
of  benefit/cost, 
assumes  benefit  for 
cost  tradeoff  rate 


Min  Fuzzy 
Pareto  K 


Algorithmically  determined  as  non- 
dominated  up  to  some  fraction  1K1  of 
objective  range 


mark  indicates  allowed  transition  from 


Utility-cost  efficiency 
gains 


Addresses  both 
costs  and  benefits 


Must  use  proper 
“costs”  (e.g, 
execution  cost) 


1*1 


iJJc 


<c 


4. 1.2.3  Evaluating  Potential  Changeability  Metrics 

One  of  the  resulting  conclusions  derived  in  our  metrics  investigation  was  that  a  good 
metric  for  adaptability  should  meet  certain  criteria.  These  include: 

•  Independent  of  design  space  enumeration  -  where  value  is  define  intrinsically 
rather  than  relative  to  other  designs 

•  Basis  is  universal  -  the  value  of  ‘metric  x’  is  equal  to  the  value  of  ‘metric  x’ 
regardless  of  variable  changes  (epoch,  design,  rule,  etc.) 

•  Values  both  magnitude  and  number  of  changes  in  that  both  of  these  provide  value 
but  in  different  ways.  Magnitude  value  implies  a  change  resulting  in  greater 
utility  than  another  is  more  valuable.  Number  of  changes  (counting  value) 
implies  that  having  two  change  options  is  better  than  one. 


As  can  be  seen  in  Table  3  no  single  metric  meets  all  of  the  desired  criteria  for  a  single 
valuable  changeability  metric.  Instead,  what  was  determined  was  to  use  a  set  of  metrics  to  gain 
insight  into  the  various  trade-offs  described  above  and  collectively  meeting  the  desirable  criteria. 
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Table  3.  Potential  Valuable  Changeability  Metrics  Evaluated  by  Criteria 


Metric 

Independent 

Universal 

Magnitude 

Number 

Filtered  Outdegree 

Maybe 

No 

No 

Yes 

Normalized  FOD 

Maybe 

Yes 

No 

No 

Value- Weighted  FOD 

Maybe 

No 

Maybe 

Yes 

Available  Rank  Increase 

No 

Yes 

Yes 

No 

It  was  determined  over  the  course  of  the  research  that  if  the  goal  of  changeability  is  value 
sustainment,  or  at  least  “effectively”  responding  to  perturbations,  then  in  addition  to 
changeability,  some  concept  of  “robustness”  should  also  be  considered  since  it  represents  the  “do 
nothing”  change  mechanism.  In  identifying  potentially  “interesting”  designs  for  the  analysis, 
including  such  designs  forms  a  natural  point  for  comparison  for  highly  changeable  designs. 

Table  4  above  lists  the  set  of  changeability  metrics  that  were  refined  over  the  course  of  the 
research  and  incorporated  in  the  valuation  approach  in  order  to  extract  the  valuable  changeability 
information  regarding  design  decisions,  strategies,  epochs,  and  change  mechanisms  over  a 
system  lifecycle.  These  metrics  will  be  illustrated  in  the  valuation  approach  description,  as  well 
as  the  case  applications  that  follow. 


Table  4.  Final  Set  of  Valuable  Changeability  Metrics 


Aspect  of 
Valuable 
Changeability 

Acronym 

Stands  For 

Definition 

Robustness  via 
“no  change” 

NPT 

Normalized  Pareto  Trace 

%  epochs  for  which  design  is  Pareto 
efficient  in  utility/cost 

Robustness  via 
“no  change” 

fNPT 

Fuzzy  Normalized  Pareto 
Trace 

Above,  with  margin  from  Pareto 
front  allowed 

Robustness  via 
“change” 

eNPT, 

efNPT 

Effective  (Fuzzy) 
Normalized  Pareto  Trace 

Above,  considering  the  design’s  end 
state  after  transitioning 

“Value”  gap 

FPN 

Fuzzy  Pareto  Number 

%  margin  needed  to  include  design 
in  the  fuzzy  Pareto  front 

“Value”  of  a 
change 

FPS 

Fuzzy  Pareto  Shift 

Difference  in  FPN  before  and  after 
transition 

“Value”  of  a 
change 

ARI 

Available  Rank  Increase 

#  of  designs  able  to  be  passed  in 
utility  via  best  possible  change 

Degree  of 
changeability 

OD 

Outdegree 

#  outgoing  transition  arcs  from  a 
design 

Degree  of 
changeability 

FOD 

Filtered  Outdegree 

Above,  considering  only  arcs  below 
a  chosen  cost  threshold 
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4.1.3  Valuation  Approach  for  Strategic  Changeability  (VASC) 

Given  that  a  number  of  metrics  address  various  important  aspects  of  valuating 
changeability,  an  “approach”  was  developed  to  aid  in  applying  these  metrics  in  order  to  uncover 
difficult-to-extract  information  on  valuable  changeability  for  a  design  space  and  present  it  in  an 
accessible  way  to  assist  in  decision  making.  Other  goals  included: 

•  Identify  designs  which  deliver  high  amounts  of  value  in  different  ways 
(robustness,  changeability),  and  the  operational  strategies  that  maximize  value 

•  Assess  what  change  mechanisms  deliver  the  most  value  or  are  the  most  critical  for 
some  designs  to  function  well 

•  Establish  cost/beneflt  tradeoff  for  adding/removing  changeability  from  a  design 
What  follows  is  a  summary  of  the  five  step  approach  to  valuating  changeability. 

Valuation  Approach  for  Strategic  Changeability  (VASC) 

1 .  Set  up  data  for  epoch-era  analysis 

2.  Identify  designs  of  interest 

3.  Define  rule  usage  strategies 

4.  Conduct  multi-epoch  changeability  analysis 

5.  Conduct  era  simulation  and  analysis 

4.1.3. 1  Set  Up  Data  for  Epoch-Era  Analysis 

Step  1  puts  the  case  in  question  into  the  epoch-era  framework,  allowing  for  piecewise 
consideration  of  time  in  sequences  of  constant-context  sections.  Activities  include  identifying 
input  data  (design  variables,  change  mechanisms,  stakeholder  preferences  and  desired  attributes, 
and  context  variables).  Outputs  include  design/epoch  lists,  transition  matrices,  and  Fuzzy  Pareto 
Number  for  each  design/epoch  pair. 

4.1.3.2  Identify  Designs  of  Interest 

Step  2  is  necessary  to  reduce  both  the  computation  time  and  the  difficulty  of  synthesizing 
and  grasping  the  results  of  the  approach  by  reducing  the  scope  of  our  full  attention.  Activities 
include  calculating  changeability  screening  metrics  (e.g.  Normalized  Pareto  Trace  and  Fuzzy 
Normalized  Pareto  Trace  for  value  robust  designs,  and  Filtered  Outdegree  for  highly  changeable 
designs),  and  any  other  desired  design  identification  techniques  (such  as  picking  favorite  designs 
through  reuse  or  high  performance  in  other  metrics).  Outputs  include  a  subset  of  designs  for 
further  exploration.  If  concurrent  visualization  for  comparison  is  desired,  then  the  number  of 
designs  in  this  set  should  be  on  the  order  of  5-7  for  clarity  purposes. 

4.1.3.3  Define  Rule  Usage  Strategies 

Defined  in  step  3,  the  strategy  is  the  unifying  factor  of  the  method,  specifying  the  logic 
that  interprets  the  system  condition  over  time  and  identifies  change  mechanism  options  that 
should  be  executed.  Activities  include  determining  the  set  of  possible  rule  usage  strategies, 
defining  strategies  in  terms  of  logic  for  change  mechanism  execution  in  each  epoch,  and  for  each 
design/epoch  pair,  determining  the  most  desirable  end  state  (defined  by  the  strategy),  which  is 
reachable  via  transition  rules.  Outputs  include  the  realized  end  states  and  transition  costs  for  each 
combination  of  design/epoch/strategy. 
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4.1.3.4  Conduct  Multi-Epoch  Changeability  Analysis 

In  step  4,  multi-epoch  changeability  analysis  considers  possible  situations  the  system 
could  be  used  in,  but  without  the  complication  of  time  ordering  or  time  dependence.  Activities 
include  calculating  multi-epoch  metrics,  such  as  Effective  NPT  and  Effective  Fuzzy  NPT,  Fuzzy 
Pareto  Shift,  Removal  Weakness,  and  Available  Rank  Increase.  Outputs  include  information  on 
when,  why,  and  how  designs  of  interest  are  changing  within  epochs  and  the  value  of  those 
changes,  as  well  as  identification  of  particularly  valuable  change  mechanisms  and/or  designs 
which  rely  on  a  single  mechanism  for  a  large  portion  of  their  value. 

4.1. 3.5  Conduct  Era  simulation  and  Analysis 

In  step  5,  sample  eras  give  important  lifecycle  information  on  the  designs  as  they 
perform,  change,  and  age  over  time,  as  well  as  help  identify  valuable  change  mechanisms. 
Activities  include  simulation  of  many  randomly  generated  potential  eras  for  each  design  of 
interest.  Outputs  include  change  mechanism  usage  frequency  and  likelihood,  era-level  statistics 
on  average/aggregate  utility  provided  and  design  efficiency,  and  comparison  of  strategies  and 
change  mechanism  usage  for  each  design. 

4.1.3.6  Metrics  Usage  in  VASC 

Each  of  the  metrics  developed  in  the  course  of  this  research  addresses  a  different  aspect 
of  the  conflicting  dimensions  for  valuating  changeability  highlighted  earlier.  Table  5  lists  the 
metrics  from  Table  4  and  to  which  step  in  VASC  its  use  is  most  applicable.  The  intent  for  the 
later  metrics  is  for  design  implications  (change  mechanism  investment)  and  strategic  decision 
making  (i.e.,  what  strategy  to  pursue,  and  cost  vs.  benefit  of  changeability  features). 


Table  5.  Metrics  with  Relation  to  Changeability  Aspects  and  VASC  Step 


Metric 

Robustness  vs. 
Changeable 

Short  Run  vs. 
Long  Run 

VASC  Step 

NPT,  fNPT 

robustness 

Short  run 

2:  screening 

FOD 

changeable 

Short  run 

2:  screening 

eNPT,  efNPT 

changeable/  robustness 

Short  run 

4:  multi-epoch 

FPS 

changeable 

Short  run 

4:  multi-epoch 

ARI 

changeable 

Short  run 

4:  multi-epoch 

Avg  FPN 

robustness 

Long  run 

5:  era  analysis 

Rule  usage 

changeable 

Long  run 

5:  era  analysis 

“going  rate” 

changeable/  robustness 

Long  run 

5:  era  analysis 

4.1.4  Case  Applications 

Several  case  applications  were  performed  on  existing  data  sets  in  order  to  develop  the 
metrics  and  valuation  approach,  as  well  as  validate  and  determine  the  scalability/deployability  of 
the  approach.  The  first  case  application,  X-TOS  was  used  for  development.  The  second  and  third 
case  applications  were  used  for  validation  and  deployability  testing. 
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4.1.4.1  X-TOS  Case  Study 

The  primary  purpose  of  the  application  of  the  X-TOS  case  in  this  research  investigation 
was  twofold:  (1)  to  serve  as  an  experimentation  case  to  develop  the  metrics  assessment  approach, 
and  (2)  to  test  the  various  adaptability  metrics  identified  through  empirical  investigation. 

Background.  X-TOS  is  a  proposed  particle-collecting  satellite  designed  to  sample 
atmospheric  density  in  low  Earth  orbit.  A  full  Multi-Attribute  Tradespace  Exploration  (MATE) 
study  was  performed  in  2002  to  analyze  potential  designs  for  the  system.  For  the  study,  8  design 
variables  were  mapped  into  7840  designs  with  5  utility-generating  attributes;  the  variables  and 
related  attributes  are  detailed  in  Table  6.  Additionally,  multiple- satellite  configurations  were 
tested,  but  not  included  in  the  final  tradespace  due  to  vastly  increased  cost  for  only  marginal 
increased  utility.  Changeability  was  noted  to  be  highly  desirable  in  the  X-TOS  final  report, 
because  an  unknown  parameter  (atmospheric  density,  which  the  system  was  designed  to 
measure)  had  a  large  impact  on  the  performance  of  the  satellite. 


Table  6.  X-TOS  Case  Design  and  Value  Attributes 


Design  Variable 

Directly  Associated  Attributes 

Apogee 

Lifetime,  Altitude 

Perigee 

Lifetime,  Altitude 

Inclination 

Lifetime,  Altitude,  Max  Latitude,  Time  at  Equator 

AntennaGain 

Latency 

Comm.  Architecture 

Latency 

Propulsion  Type 

Lifetime 

PowerType 

Lifetime 

AV  Capability 

Lifetime 

In  2006,  the  X-TOS  study  was  revived  as  a  case  study  for  a  research  effort  to  quantify 
changeability  (Ross  and  Hastings  2006  Error!  Bookmark  not  defined.).  Using  the  change 
agent-mechanism-effect  framework,  8  transition  rules  were  created  allowing  for  the  change  from 
one  design  point  to  another.  The  rules  are  listed  in  Table  7.  It  should  be  noted  that  the  “tugable” 
and  “refuelable”  designations  were  added  as  design  variables,  to  be  included  at  a  fixed  cost  as 
enablers  for  the  appropriate  change  mechanisms.  The  study  encompasses  Adaptability  (i.e., 
SEAri  adaptability  and  flexibility),  as  the  “bum  fuel”  change  mechanism  responds  to  an  internal 
agent  (adaptable),  while  the  others  respond  to  external  agents  (flexible).  There  is  also  an 
appreciable  spread  in  costs,  as  the  Bum  Fuel  mechanism  is  modeled  to  cost  orders  of  magnitude 
less  to  employ,  in  both  time  and  money,  than  the  others. 
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Table  7.  X-TOS  Rules  (Change  Mechanisms) 


Rule 

Description 

Change  agent  origin 

Rl:  Plane  Change 

Increase/decrease  inclination,  decrease  AV 

Internal  (Adaptable) 

R2:  Apogee  Bum 

Increase/decrease  apogee,  decrease  AV 

Internal  (Adaptable) 

R3 :  Perigee  Bum 

Increase/decrease  perigee,  decrease  AV 

Internal  (Adaptable) 

R4:  Plane  Tug 

Increase/decrease  inclination,  requires  “tugable” 

External  (Flexible) 

R5 :  Apogee  Tug 

Increase/decrease  apogee,  requires  “tugable” 

External  (Flexible) 

R6:  Perigee  Tug 

Increase/decrease  perigee,  requires  “tugable” 

External  (Flexible) 

R7:  Space  Refuel 

Increase  AV,  requires  “refuelable” 

External  (Flexible) 

R8:  Add  Sat 

Change  all  orbit,  AV 

External  (Flexible) 

Finally,  58  different  epochs  were  generated  to  extend  the  study  into  epoch-era  analysis. 

In  this  case,  the  epochs  were  generated  by  perturbing  the  defined  stakeholder  utility  preferences 
of  the  MATE  study  in  one  of  four  ways:  adding/removing  attributes,  reweighting  the  attributes, 
linearizing  the  attribute  utility  curves,  and  altering  the  multi-attribute  utility  aggregation 
function.  These  epochs  form  a  basis  for  a  “what  if?”  analysis  of  the  future,  addressing  such 
design  process  uncertainties  as  “What  if  we  selected  the  wrong  attribute  set?”  or  “What  if  the 
stakeholder  changes  preferences?”).  This  is  an  acceptable  method  of  generating  epochs, 
although  ideally  the  stakeholder  would  be  available  to  re-derive  his  preferences  for  different 
potential  scenarios  in  the  system’s  future  such  as  a  goal  change  or  wartime  conditions. 

Step  1:  Set  Up  Data  for  Epoch-Era  Analysis 

Initiating  the  method  requires  the  proper  construction  of  an  epoch-differentiated  data  set. 
To  reiterate,  epochs  are  defined  as  periods  of  time  during  which  the  system  operates  in  a 
particular  fixed  context  and  set  of  needs.  An  epoch  is  defined  by  a  set  of  epoch  variables,  which 
must  be  enumerated.  Epoch  variables  should  include  any  and  all  situational  or  operational 
conditions  that  will  change  over  time  and  significantly  impact  the  system’s  delivery  of  value. 
The  epochs  must  differentiate  the  candidate  designs  in  value  or  the  results  of  the  method  will  be 
simply  a  repetitious  version  of  a  static  context  study;  for  this  reason,  selecting  value-affecting 
epoch  variables  is  a  critical  step  towards  investigating  the  value-over-time  characteristics  of 
different  designs.  Epochs  are  differentiated  by  varying  stakeholder  preferences  in  four  ways:  (1) 
Changing  value-delivering  attribute  set;  (2)  Changing  attribute  weightings  in  multi-attribute 
utility  function;  (3)  Linearizing  attribute  utility  curves;  and  (4)  Different  utility  aggregating 
functions.  The  epochs  are  enumerated  one  at  a  time  as  perturbations  from  the  base  case  are 
never  applied  simultaneously.  These  epochs  represent  anticipatory  exploration  of  possible 
preferences,  helping  to  answer  “what  if’  questions,  such  as: 

•  What  if  you  don’t  elicit  the  “right”  requirements/preferences  (attributes)? 

•  What  if  you  don’t  elicit  the  “right”  attribute  priorities? 

•  What  if  you  don’t  elicit  the  “right”  utility  curve  shape? 
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More  specific  epochs  can  be  created  by  actually  re-deriving  the  utility  curves  with  the 
stakeholder  for  different  hypothetical  situations  (mission  goal  change,  wartime,  etc.) 

Step  2:  Identify  Designs  of  Interest 

It  is  useful  to  use  a  set  of  screening  metrics  in  order  to  reduce  the  number  of  designs 
considered  in  full  detail  to  a  manageable  amount  (on  the  order  of  10  instead  of  the  entire 
tradespace).  One  such  metric  is  Normalized  Pareto  Trace  (NPT),  which  identifies  designs  that 
are  passively  value  robust:  cost-utility  efficient  in  a  large  number  of  epochs.  NPT  is  plotted  for 
the  entire  space  in  Figure  17,  with  Design  31  highlighted  as  the  best-NPT  design.  Design  31  is 
thus  a  candidate  for  further  investigation,  as  it  might  be  expected  to  perform  well  based  on  its 
robustness.  By  considering  Fuzzy  Normalized  Pareto  Trace  (fNPT),  designs  that  are  nearly  cost- 
utility  efficient  will  also  be  identified:  Figure  18  shows  this  plot  and  highlights  Designs  1,  345, 
689,  and  2759.  Note  that  these  are  not  the  only  designs  with  an  fNPT  of  1 ;  they  were  selected  to 
provide  as  broad  a  range  of  the  design  variables  as  possible.  If  VASC  is  iterated  a  second  time 
on  other  designs  of  interest,  there  are  many  options  here. 


All  designs  --  NPT 
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x=i  x=34s  x=689AII  designs  -- 1%  fNPT 

Y  =  1  Y  =  1  Y  =  1  ° 


Another  useful  screening  metric  is  Filtered  Outdegree  (FOD),  which  can  be  used  to 
identify  designs  with  a  large  number  of  outgoing  change  arcs.  Heuristically,  these  designs  are 
expected  to  derive  more  value  from  changeability  because  of  their  increased  number  of  options; 
this  will  be  tested  with  the  rest  of  the  VASC  process.  Varying  the  filter  in  the  FOD  equation 
allows  designs  to  be  found  with  large  numbers  of  options  available  for  different  levels  of 
acceptable  transition  cost;  this  can  be  useful  because  a  design  with  a  large  number  of  cheap 
transition  options  but  not  the  most  options  overall  may  provide  a  less  expensive  alternative. 
Figure  19  shows  the  FOD  for  the  design  space  at  two  different  thresholds:  one  essentially 
unlimited  (1010  dollars  and  seconds)  and  the  other  quite  limited  (103  dollars  and  105  seconds), 
identifying  four  more  designs  of  interest. 
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Figure  19.  X-TOS  -  Design  Space  FOD  for  Two  Thresholds 


The  selected  designs  of  interest  are  listed  in  Table  8  with  the  respective  values  of  their 
design  variables  and  a  reference  letter  to  be  used  in  future  plots.  Note  that  the  screening  metrics 
selected  only  designs  with  chemical  propulsion:  this  is  a  first-order  insight  suggesting  that 
chemical  propulsion  is  dominant  over  the  other  options  (in  this  case,  electric).  If  the  decision 
maker  feels  that  electric  propulsion  carries  some  positive  characteristic  which  is  not  being 
captured  by  the  utility  function  or  the  screening  metrics,  VASC  could  be  repeated  with  a 
different  set  of  designs  of  interest,  focusing  on  electric  satellites.  Also  of  interest  is  the  fact  that 
three  of  the  designs  identified  here  match  the  set  of  designs  of  interest  used  in  the  2006  X-TOS 
changeability  study,  suggesting  that  the  screening  metrics  are  in  line  with  previous  expert 
insights. 
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Table  8.  X-TOS  Case  Study  Designs  of  Interest 


Design  £1 

Reference 

Inclination 

ApogEe 

Perigee 

Delta  V 

Prop  Type 

Powet  Type 

Ant.  Gain 

1 

A 

30 

458.33 

150 

1200 

Chem 

Fuel 

Low 

31 

B 

30 

458.33 

233.33 

1200 

Chem 

Fuel 

Low 

176 

C 

30 

1075 

350 

1200 

Chem 

Solar 

High 

345 

D 

70 

458.33 

150 

1200 

Chem 

Fuel 

Low 

403 

E 

70 

458.33 

350 

100 

Chem 

Solar 

Hi:gh 

6S9 

F 

00 

458.33 

150 

1200 

Chem 

Fuel 

Low 

752 

G 

00 

458.33 

350 

too 

Chem 

Solar 

Hiigh 

364 

H 

00 

1075 

350 

1200 

Chem 

Solar 

Hugh 

2759 

t 

00 

766.67 

216.67 

400 

Chem 

Fuel 

Low 

Step  3:  Define  Rule  Usage  Strategies 

For  X-TOS,  two  strategies  are  considered:  (1)  maximize  utility  and  (2)  maximize 
efficiency  (as  measured  by  Fuzzy  Pareto  Number,  FPN).  These  are  basic  strategies  that  target 
simple  measures  of  system  performance  at  any  cost,  but  serve  well  to  predict  how  a  stakeholder 
may  choose  to  use  the  system. 

Maximize  Utility:  Make  system  as  good  at  its  job  as  possible  (highest  reachable  utility 
per  epoch). 

Maximize  Efficiency:  Desire  to  be  as  cost-utility  efficient  as  possible. 

After  picking  the  strategies,  a  MATLAB®  script  finds  the  executed  transition  (if  any)  for 
each  design  in  each  epoch  with  each  strategy,  which  feeds  into  the  following  steps. 

Step  4:  Conduct  Multi-Epoch  Changeability  Analysis 

Table  9  shows  the  NPT  of  the  designs  of  interest  side  by  side  with  the  Effective 
Normalized  Pareto  Trace  (eNPT)  resulting  from  each  strategy.  Maximizing  efficiency  clearly 
results  in  an  increase  from  NPT  to  eNPT  for  each  design  because  the  strategy  does  not  allow  for 
changes  that  move  designs  away  from  the  Pareto  Front.  The  maximize  utility  strategy  results  in  a 
decrease  for  most  designs,  as  greater  utility  can  be  achieved,  but  with  diminishing  returns  on 
costs,  thus  reducing  efficiency  and  moving  away  from  the  Pareto  Front. 
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Table  9.  X-TOS  -  NPT  and  eNPT 


Design 

Do  Nothing 
(NPT)  ' 

Max  U 

Max  Eff 

A 

0.776 

0.103 

1 

B 

0.810 

0.103 

1 

C 

0 

0 

0,052 

D 

0.017 

0.086 

1 

E 

0 

0 

0.052 

F 

0.207 

0.086 

1 

G 

0 

0 

0.052 

H 

0 

0 

0.052 

! 

0 

0 

1 

Table  10  shows  the  corresponding  fNPT  and  Effective  Fuzzy  Normalized  Pareto  Trace 
(efNPT)  of  the  designs  of  interest  with  a  1%  fuzziness,  and  improvements  over  the  “not-fuzzy” 
values  highlighted  in  green.  Obviously,  all  of  the  designs  of  interest  perform  well  in  these 
metrics  across  most  epochs  when  allowed  to  execute  transitions  and  when  allowing  for  a  small 
margin  of  fuzziness  in  defining  efficiency. 


Table  10.  X-TOS  -  1%  fNPT  and  efNPT 


Design 

Do  Nothing 
(fNPT)  ’ 

Max  U 

Max  Eff 

A 

1 

1 

B 

0.948 

i 

1 

C 

0 

0.879 

0.914 

D 

1 

1 

1 

E 

0 

0.879 

0.914 

F 

1 

1 

|  1 

G 

0 

0.879 

0.914 

H 

0 

0.879 

0.914 

1 

1 

1 

1 

Table  1 1  and  Figure  20  below  show  the  FPS  distributions  and  order  statistics  for  the 
designs  of  interest  under  the  maximize  utility  strategy.  Comparing  the  Fuzzy  Pareto  Shift  (FPS) 
distributions  of  the  designs  of  interest  is  the  main  focus  of  this  step  of  VASC,  as  it  allows  for  an 
understanding  of  the  similarities  and  differences  between  the  designs.  Designs  A  and  B,  which 
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were  high  NPT  selections,  obviously  do  not  transition  frequently,  as  visible  in  the  large  spike  at 
zero  FPS,  although  B  does  have  a  +40  FPS  for  one  epoch  in  which  it  performs  poorly.  Designs 
C  and  H  appear  to  derive  the  most  value  from  their  changeability  under  this  strategy;  each  has  a 
spike  in  the  low  twenties  range  that  comprises  about  half  of  the  epochs  in  the  epoch  space,  with 
no  epochs  causing  a  reduction  in  efficiency  and  a  few  high  outliers  over  sixty.  E  and  G  also 
perform  well,  with  consistent  improvement  of  around  7%  efficiency  in  most  epochs.  The  table 
provides  a  slightly  less  cluttered  view  of  the  same  information,  with  the  results  highlighted  by  a 
heat  map  from  red  (bad)  to  green  (good).  Note  that  order  statistics  are  presented  and  not  mean  or 
standard  deviation:  since  FPS  distributions  are  frequently  skewed  (as  they  are  here),  the  mean  is 
a  misleading  measure  of  central  tendency. 


Table  11.  X-TOS  -  Maximize  Utility  FPS  Statistics 


Min  IstQ  Med  3rd  Q  Max 


A 

-1 

-1 

-1 

0 

1 

B 

-I 

-1 

-1 

0 

40 

C 

2 

25 

2G 

26 

65 

D 

0 

0 

0 

0 

I 

E 

4 

7 

7 

7 

7 

F 

-1 

0 

0 

0 

I 

G 

4 

7 

7 

7 

7 

H 

2 

22 

25,5 

26 

65 

1 

0 

0 

0 

0 

I 

Figure  20.  X-TOS  -  Maximize  Utility  FPS  Distribution 


Table  12  and  Figure  21  show  the  same  information  but  for  the  maximize  efficiency 
strategy.  The  designs  perform  largely  similarly  to  the  maximize  utility  strategy,  with  the 
exception  of  the  few  negative  performances  becoming  zeros  because  this  strategy  does  not  allow 
for  negative  FPS  changes. 
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Table  12.  X-TOS  -  Maximize  Efficiency  FPS  Statistics 


Dssign  Valuable  Changeability  Frequence 
GO 


Mtn  1st  Q  MeiJ  3rd  Q.  Max 


A 

B 

0 

0 

0 

0 

0 

0 

0 

J> 

1 

m 

C 

2 

25 

26 

26 

65 

D 

1 

1 

1 

I 

E 

6 

7 

7 

7 

S 

F 

1 

1 

i 

1 

G 

6 

7 

7 

7 

a 

H 

2 

22 

26 

26 

65 

1 

1 

1 

1 

1 

1 

Figure  21.  X-TOS  -  Maximize  Efficiency  FPS  Distribution 


Step  5:  Conduct  Era  Simulation  and  Analysis 

A  simple  era  constructor  was  created  in  MATLAB®  for  the  purposes  of  this  case  study. 
The  era  constructor  rules  are  defined  as  follows: 

1 .  20  randomly  selected  epochs 

2.  Each  epoch  is  1  year  in  duration 

This  is  a  very  simple  era  constructor,  but  it  encompasses  the  basic  features  desired  by  this 
particular  study.  Since  the  epochs  vary  only  by  preference  set,  the  goal  of  era  analysis  is  to 
understand  the  performance  effects  of  uncertain  varying  preferences  over  time  (the  long  run) 
and,  in  particular,  the  use  of  the  change  mechanisms.  This  makes  features  such  as  varying  epoch 
lengths  or  more  sophisticated  sampling  of  epochs  unimportant,  although  they  could  be 
implemented  as  well. 

One  thousand  eras  were  constructed  and  analyzed  for  each  candidate  design  in  the  study. 
As  the  trials  were  running,  the  total  dollar  cost,  time  cost,  and  number  of  changes  were  recorded, 
to  be  averaged  at  the  end,  along  with  tracking  of  FPN  across  the  era  and  transition  usage  by 
change  mechanism.  That  data  is  presented  in  Table  13,  and  it  reveals  a  number  of  interesting 
outcomes. 
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Table  13.  X-TOS  -  Era  Analysis  Results 


Max  Utility  Max  Efficiency 


Design 

Avg  # 
Trans. 

Avg  Total 
Trans 

Avg  Total 
Trans 

Avg  FPN 

Design 

Avg  # 
Trans. 

Avg  Total 
Trans 

Avg  Total 
Trans 

Avg  FPN 

Cost 

Delay 

Cost 

Delay 

A 

19.9 

$386M 

2.91  yrs 

2.76 

A 

7.5 

$27 1M 

2.97  yrs 

1.99 

B 

19.9 

$382M 

2,86  yrs 

2.73 

B 

7.1 

$260M 

2.S5  yrs 

2.21 

C 

19.8 

$396M 

2.96  yrs 

5.54 

C 

10.3 

$31 9M 

3.44  yrs 

5.18 

D 

19.9 

$422 M 

3,34  yrs 

2.69 

D 

8.3 

$31 0M 

3.38  yrs 

2.22 

E 

19.9 

$432M 

3.42  yrs 

4.47 

E 

10.2 

S335M 

3.66  yrs 

4.19 

F 

19.8 

$420 M 

3  32  yrs 

2.68 

F 

8.0 

$300M 

3.25  yrs 

2.23 

G 

19.8 

$425M 

3.33  yrs 

4.68 

G 

10.2 

$329M 

3.59  yrs 

4.17 

H 

19.8 

$419M 

3.25  yrs 

5.47 

H 

10.1 

$306M 

3.30yrs 

5.06 

1 

19.9 

$422 M 

3,34  yrs 

2.64 

1 

8.4 

$314M 

3.41  yrs 

2.28 

First,  it  is  obvious  that  maximizing  utility  requires  a  transition  at  nearly  every  epoch 
switch.  This  makes  sense  considering  that  the  epochs  define  different  preferences  and  thus, 
assuming  that  the  preferences  change  enough  to  differentiate  themselves,  each  epoch  will  have  a 
different  best-utility  design  reachable  by  the  system.  It  is  also  apparent  that,  despite  fewer 
transitions  and  lower  amounts  of  money  spent  on  transitions,  the  maximize  efficiency  strategy  has 
approximately  the  same  amount  of  time  delay  from  transitions  (3  out  of  20  years,  ~15%). 

Finally,  by  tracking  FPN  across  the  era,  we  can  take  the  average  FPN  as  a  measure  of  the 
lifetime  cost  efficiency  of  the  system,  which  varies  between  2%  and  5%  inefficient  for  the 
different  designs  of  interest:  a  relatively  small  variation.  Thus,  it  appears  that  the  designs  are  all 
quite  similar  in  operation  given  the  changeability  strategies  used  here,  with  a  slight  advantage  to 
the  maximize  efficiency  strategy  and  passively  robust  designs  (A,B)  for  their  lower  expected 
transition  costs. 

Identifying  why  the  designs  have  such  similar  performances  is  potentially  interesting.  By 
looking  at  the  transitions  selected  by  each  strategy,  likelihoods  for  using  each  change  mechanism 
for  a  random  epoch  switch  can  be  calculated.  These  are  presented  in  Table  14.  The  two  most 
common  mechanisms  are  Perigee  Burn  and,  surprisingly,  Redesign.  Perhaps  this  sort  of 
behavior  is  not  what  is  desired  or  expected  in  the  system;  the  stakeholder  may  want  to  find  a 
design  that  is  functional  over  an  era  without  needing  to  redesign.  To  address  this,  a  rule  removal 
weakness  study  can  be  performed. 
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Table  14.  X-TOS  -  Change  Mechanism  Usage  Likelihood  (Random  Epoch  Switch) 


Inclinetic-n 

Apogee 

Perigee 

Inclination  Apogee 

Perigee 

Design 

Burn 

Bunn 

Burn 

Tug 

Tug 

Tug 

Refuel 

Redesign 

A 

0 

0.02 

0.93 

0 

0 

0.05 

0 

0.17 

B 

0 

0.02 

0.93 

0 

0 

0.02 

0 

0.17 

C 

0.17 

0.72 

0 

0 

0  10 

0.79 

0 

0.17 

Max  Utility 

D 

E 

0 

0 

0 

0.02 

1.00 

0.86 

0 

0 

0 

0 

0 

0 

0 

0 

1.00 

1.00 

F 

0 

0 

1.00 

0 

0 

0 

0 

0.34 

G 

0 

0.02 

0.36 

0 

0 

0 

0 

1.00 

H 

0.79 

0.19 

0 

0 

0.02 

0.17 

0 

0.63 

1 

0 

0.02 

0.90 

0 

0 

0 

0 

1.00 

inclination 

Apogee 

Perigee 

Inclination 

Apogee 

Perigee 

Design 

Bum 

Bum 

Bum 

Tug 

Tug 

Tug 

ReFuel 

Redesign 

A 

D 

0 

0.22 

0 

0 

0 

0 

0.16 

B 

0 

0 

016 

0 

0 

0.03 

0 

0.16 

C 

0.47 

0.45 

0 

0 

0.09 

0.41 

0 

0.55 

Max  Efficiency 

0 

E 

0 

0 

0 

0.02 

0.93 

0.33 

0 

0 

0 

0 

0 

0 

0 

0 

0.98 

1.00 

F 

0 

0 

0.79 

0 

0 

0 

0 

0.79 

G 

0 

0.02 

0.33 

0 

0 

0 

0 

1.00 

FI 

0.41 

0.57 

0 

0 

0.02 

0.47 

0 

0.47 

i 

0 

0.10 

0.79 

0 

0 

0 

0 

1.00 

To  perform  a  removal  weakness  study,  the  strategies  must  be  reevaluated  without 
considering  any  transition  arcs  that  utilize  the  removed  rule.  This  allows  the  criticality  of  that 
rule  to  be  evaluated  for  each  design  of  interest.  Then,  the  removal  weakness  is  calculated  as  the 
difference  in  FPS  caused  by  the  removal,  and  can  be  plotted  in  a  distribution  as  in  Figure  22. 
Some  designs  (C,F)  have  no  change,  but  most  have  ~2%  decrease  in  efficiency  in  most  epochs, 
with  worst  cases  approaching  -12%  for  Maximize  Utility  and  -6%  for  Maximize  Efficiency. 


Design  Vt-jJDte  Changeability  FttHfjftrcies 

60  r- 


Max  Utility 


Max  Efficiency 


Figure  22.  X-TOS  -  Removal  Weakness  (FPS  Impact  of  Removing  “Redesign”) 
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Eras  can  also  be  re-run  with  the  new  strategic  transitions  to  see  era-level  effects  of  the 
removal  of  redesign.  These  statistics  are  shown  in  Table  15.  It  is  apparent  that,  while  number  of 
transitions  and  average  FPN  are  about  the  same,  the  dollar  and  time  costs  of  transitions  have 
changed  dramatically.  Transition  time  delay  has  decreased  from  years  to  days,  as  expected  from 
removing  the  redesign  cycle,  dramatically  increasing  the  amount  of  time  for  which  the  system  is 
active;  this  could  be  very  attractive  to  a  stakeholder  if  this  were  a  revenue-generating  project  or  if 
utility-months  was  the  lifetime  value  metric  of  choice  rather  than  average  FPN.  However, 
transition  costs  have  gone  up  about  an  order  of  magnitude  as  well  (this  is  because  the  modeling 
of  redesign  transition  costs  does  not  include  the  relaunch  cost:  more  careful  modeling  of  that 
would  allow  more  accurate  cost  comparisons). 


Table  15.  X-TOS  -  Era  Statistics  (Redesign  Removed) 
Max  Utility  Max  Efficiency 


Design 

Avg  # 
Trans. 

Avg  Total 
Trans 

Avg  Total 
Trans 

Avg  FPN 

Design 

Avg  # 
Trans. 

Avg  Total 
Trans 

Avg  Total 
Trans 

Avg  FPN 

Cost 

Delay 

Cost 

Delay 

A 

19.9 

S4.02B 

9.5  days 

271 

A 

7.4 

$3.59B 

8.4  days 

1.79 

B 

19.9 

S4.08B 

9.5  days 

2.68 

B 

7.1 

S3.55B 

8,2  days 

1.96 

C 

19.3 

$3.97B 

11.0  days 

5.69 

C 

10.4 

S4.00B 

12.4  days 

5.56 

D 

19.9 

S4.57B 

10.3  days 

2.86 

D 

8.2 

$3.92B 

9.6  days 

2.06 

E 

19.9 

$4.41  B 

12.2  days 

4.51 

E 

10.8 

$4.51 B 

13.9  days 

4.57 

F 

19.3 

$4. SOB 

10.3  days 

277 

F 

8. 

S4.18B 

9.5  days 

1.97 

G 

19.3 

$4.51  B 

12.1  days 

4.56 

G 

10.4 

S4.32B 

13.3  days 

4.47 

H 

19.8 

$4.39B 

11.6  days 

5.55 

H 

10.6 

S4.25B 

12.7  days 

5.42 

1 

19.9 

$4.26B 

12.7  days 

2.89 

1 

7.7 

$3,49B 

10.0  days 

1.99 

As  a  final  synthesis,  consider  that  the  stakeholder  has  selected  Design  B  (31)  for  its 
combination  of  high  fNPT  and  high-scoring  FPS  in  the  few  epochs  in  which  it  performs  poorly, 
but  is  interested  in  potential  tweaks  to  the  design.  Looking  back  at  Table  14,  it  is  obvious  that  B 
never  utilizes  a  refuel  and  almost  never  utilizes  the  tug  feature.  If  it  is  possible  to  isolate  and 
remove  these  features,  this  represents  a  potential  means  to  reduce  design  costs  of  Design  B.  This 
benefit  of  reduced  costs  comes  at  a  penalty  of  slightly  reduced  changeability  (by  removing  the 
few  times  you  would  choose  to  execute  a  tug)  that  can  be  quantified  by  another  removal 
weakness  study.  Alternatively,  if  another  potential  change  mechanism  has  been  deemed  feasible 
(for  example,  a  variable  angle  sampling  scoop),  additional  modeling  could  be  used  to  estimate  its 
cost,  and  it  can  be  inserted  into  the  study  to  calculate  its  lifetime  performance  benefits. 

Modeling  efforts  like  this  can  be  used  to  establish  a  “going  rate”  for  changeability  in  the  system: 
the  cost/benefit  tradeoff  of  adding  or  removing  changeability  from  the  selected  design. 
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Case  Outcome.  As  a  result  of  the  experimentation  case,  the  team  modified  the  approach 
and  clarified  the  steps,  as  well  as  refined  the  metrics.  The  result  was  the  Valuation  Approach  for 
Strategic  Changeability  (VASC)  and  supporting  metrics,  which  were  subsequently  applied  to  the 
Space  Tug  Case  and  the  Satellite  Radar  Case. 

4.1.4.2  Space  Tug  Case  Study 

The  primary  purpose  of  the  application  of  the  Space  Tug  case  in  this  research 
investigation  was  to  demonstrate  the  end-to-end  process  in  a  relatively  simple  case.  In  particular, 
the  application  to  the  Space  Tug  system  demonstrates  both  evaluation  of  valuable  changeability 
within  epochs  (short  run  value  of  changeability)  as  well  as  across  eras  via  “strategies”  (long  run 
value  of  changeability). 

Background.  A  space  tug  is  a  vehicle  designed  to  rendezvous  and  dock  with  a  space 
object;  make  an  assessment  of  its  current  position,  orientation,  and  operational  status;  and,  then, 
either  stabilize  the  object  in  its  current  orbit  or  move  the  object  to  a  new  location  with 
subsequent  release.  A  previous  MATE  study  explored  the  tradespace  for  a  general-purpose 
servicing  vehicle  (McManus  and  Schuman,  2003).  Three  attributes  formed  the  multi-attribute 
utility  function:  total  AV  capability,  capability  of  the  grappling  system,  and  response  time  (slow 
or  fast).  To  provide  these  attributes,  three  design  variables  were  considered  in  subsequent 
modeling  activities:  manipulator  mass,  propulsion  type,  and  fuel  load.  A  full-factorial,  design 
space  was  sampled  and  analyzed — featuring  128  designs — by  inputting  each  possible 
combination  of  design  variables  from  a  set  of  enumerated  values  over  a  range  into  (1)  a 
parametric  cost  estimation  model  and  (2)  a  physics-based  performance  model. 

Step  1:  Set  Up  Data  for  Epoch-Era  Analysis 

In  order  to  apply  the  Space  Tug  dataset  for  this  analysis,  the  original  three  design 
variables  were  expanded  to  four  design  variables,  which,  when  enumerated,  resulted  in  384 
designs.  The  design  variables  were: 

•  Propulsion  type  (biprop,  cryo,  electric,  or  nuclear) 

•  Fuel  mass 

•  Capability  level 

•  Design  for  changeability  (DFC)  level 

The  DFC  level  is  a  switch  intended  to  model  a  conscious  effort  to  design  for  ease  of 
redesign/change.  In  the  model,  it  varies  from  0  to  1  to  2,  with  the  reward  of  additional  and/or 
cheaper  change  mechanisms,  and  the  penalty  of  additional  dry  mass,  resulting  in  higher  costs  and 
lower  available  deltaV. 

In  addition  to  the  design-space,  there  were  16  epochs  considered,  generated  from  2 
contexts  and  8  user  preference  sets.  The  2  contexts  corresponded  to  present  or  future  technology 
level,  which  affects  the  transition  costs,  fuel  efficiencies,  and  mass  fractions. 
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In  order  to  generate  the  tradespace  network  for  Space  Tug,  six  change  mechanisms  were 
defined  and  are  listed  in  Table  16.  Rules  1-5  are  “redesign”  rules,  which  require 
decommissioning  and  relaunching  a  space  tug  (with  the  associated  costs)  and  rule  6  is  an 
“operations”  rule,  and  does  not  require  a  new  space  tug. 


Table  16.  Space  Tug  Study  Transition  Rules  (Change  Mechanisms) 


# 

Rule 

Effect 

DFC  level 

1 

Engine  Swap 

Biprop  < — >cryo 

0 

2 

Fuel  Tank  Swap 

Change  propellant  mass 

0 

3 

Engine  Swap  (reduced  cost) 

Biprop  < — >cryo 

1  or  2 

4 

Fuel  Tank  Swap  (reduced  cost) 

Change  propellant  mass 

1  or  2 

5 

Change  capability 

Change  capability 

1  or  2 

6 

Refuel  in  orbit 

Change  propellant  mass 
(no  redesign) 

2 

Once  these  rules  were  defined,  an  automated  algorithm  determined  the  accessibility  of 
each  design  in  the  tradespace  (N=384)  to  one  another  via  each  of  the  6  transition  rules,  along 
with  calculating  the  transition  cost  (dollars  and  time)  for  each  allowed  path  between  two  designs. 
After  these  transition  matrices  were  calculated,  a  multi-arc  calculation  was  performed  to 
determine  the  “non-dominated”  paths  linking  any  two  designs  in  the  tradespace.  The  “collapsed” 
transition  matrix  lists  the  most  efficient  (in  terms  of  cost  and  time)  paths  allowed  between  any 
two  designs  in  the  tradespace.  The  multi-arc  transition  matrix  is  illustrated  with  a  “spyplot”  in 
Figure  23,  with  each  mark  indicating  allowable  transition  from  row  i  to  column  j. 
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Figure  23.  Space  Tug  Multi-arc  Transition  Matrix 


Step  2:  Identify  Designs  of  Interest 

After  setting  up  the  data  for  epoch-era  analysis,  the  next  step  is  to  identify  designs  of 
interest.  For  purposes  of  VASC,  “interesting”  designs  are  those  that  have  a  high  likelihood  of 
being  valuable  over  a  period  of  time,  such  as  the  intended  lifecycle  for  a  system.  Two  categories 
of  potentially  interesting  designs  include  those  that  are  “passively  value  robust”  and  those  that 
are  highly  changeable.  The  former  designs  perform  well  across  a  number  of  epochs  without 
needing  to  change.  The  latter  designs  have  a  large  “degree”  of  change,  but  it  is  unknown  if  the 
accessible  end  states  are  of  any  value. 

In  order  to  identify  the  “passively  value  robust”  designs,  the  Normalized  Pareto  Trace 
(NPT)  and  Fuzzy  Normalized  Pareto  Trace  (fNPT)  can  be  used,  with  high  scores  indicating 
“interesting”  designs  for  further  consideration.  NPT  can  be  calculated  by  counting  the  fraction  of 
epochs  in  which  a  given  design  appears  in  the  utility-cost  Pareto  set  (Figure  24).  fNPT  is 
calculated  by  allowing  the  definition  of  “Pareto  set”  to  include  designs  within  K%  of  the  Pareto 
Frontier.  For  this  study,  the  1%  and  15%  fuzzy  Pareto  Frontier  was  used  (Figure  25).  In  order  to 
identify  the  highly  changeable  designs,  the  Filtered  Outdegree  (FOD)  can  be  calculated  by 
counting  the  number  of  accessible  end  states  available  for  a  given  starting  design  state.  The  filter 
is  a  constraint  on  the  amount  of  dollars  and  time  (transition  cost)  willing  to  be  spent  in  executing 
a  change.  As  the  filter  becomes  more  constraining,  the  FOD  decreases  differentially  across 
design  alternatives.  No  filter  results  in  counting  all  accessible  end  states,  regardless  of  transition 
costs,  which  is  the  Outdegree  (OD)  of  a  design  in  the  tradespace  network.  For  this  study,  both  no 
filter,  and  a  four  month  transition  time  filter  were  applied  (Figure  26). 
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Fuzzy  NPT  (1%) 


Design  Number 


Figure  24.  Space  Tug  Designs  with  High  NPT 


Identifying  designs  with  high  fNPT 
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Figure  25.  Space  Tug  Designs  with  High  fNPT 
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FOD  <no  filter) 


Identifying  designs  with  high  FOD 


Design  Number  Design  Number 

Figure  26.  Space  Tug  Designs  with  High  FOD 


The  three  figures  above  illustrate  the  screening  metrics  across  the  design  space  and  the 
indicated  chosen  designs  of  interest.  Table  17  summarizes  the  selected  designs  of  interest  after 
applying  the  screening  metrics. 


Table  17.  Space  Tug  Designs  of  Interest 


Design 

Number 

Ref 

Prop 

Type 

DFC 

Level 

Fuel 

Mass 

(kg) 

Capability 

(kg) 

Fast? 

DeltaV 

(m/s) 

Base 

Cost 

($M) 

1 

A 

Biprop 

0 

30 

300 

Y 

143 

97 

29 

B 

Nuke 

0 

1200 

300 

Y 

7381 

306 

47 

C 

Cryo 

0 

10000 

1000 

Y 

6147 

628 

128 

D 

Nuke 

0 

30000 

5000 

Y 

14949 

3020 

191 

E 

Nuke 

1 

10000 

1000 

Y 

16150 

980 

328 

F 

Biprop 

2 

50000 

3000 

Y 

4828 

2804 

376 

G 

Elec 

i 

2 

30000 

5000 

i 

N 

27829 

_ 1 

3952 

- y — ■ - 

Design  variables  Design  attributes  (promt  context) 
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Step  3:  Define  Rule  Usage  Strategies 

Following  the  selection  of  designs  of  interest,  the  next  step  is  to  define  the  potential  rule 
usage  strategies,  which  will  be  used  to  select  the  “best”  end  states  for  each  design/epoch  pair. 
The  strategies  used  in  the  Space  Tug  analysis  are  described  below. 

Maximize  Utility:  Make  system  as  good  at  its  job  as  possible  (highest  reachable  utility 
per  epoch). 

Maximize  Efficiency:  Desire  to  be  as  cost-utility  efficient  as  possible. 

Survive:  Execute  change  only  if  system  risks  becoming  “invalid.” 

Maximize  Profit:  (Given  a  revenue  model)  use  design  changes  to  maximize  revenues 
less  costs  in  each  epoch. 

Step  4:  Conduct  Multi-Epoch  Changeability  Analysis 

The  next  step  in  the  approach  is  to  conduct  multi-epoch  analysis,  that  is,  conduct  analysis 
across  the  various  potential  epochs  to  see  the  distribution  of  valuable  changeability  across 
possible  alternative  future  context-needs  pairs. 

One  of  the  activities  in  this  step  is  the  calculated  of  the  Effective  NPT  (eNPT)  and  the 
Effective  Fuzzy  NPT  (efNPT).  These  metrics  are  calculated  in  a  similar  to  NPT  and  fNPT, 
however,  instead  of  only  considering  the  originating  design  state,  this  calculation  looks  at  the 
“best”  end  state  in  a  given  epoch.  (Recall  that  given  a  strategy,  there  is  one  “best”  end  state  in 
that  epoch.) 

The  “do  nothing”  strategy  is  included  for  comparison  and  is  equivalent  to  the  “robust” 
design  approach,  where  no  change  mechanism  will  be  executed.  Figure  27  illustrates  the  impact 
of  strategy  on  efNPT  across  the  seven  designs  of  interest. 
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Exploring  Effective  Fuzzy  Normalized 
Pareto  Trace  for  each  strategy 
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Figure  27.  Space  Tug  efNPT  by  Strategy  for  Designs  of  Interest 

Fuzzy  Pareto  Shift  (FPS)  can  likewise  be  calculated  to  see  how  much  the  Fuzzy  Pareto 
Number  (FPN,  a  measure  of  “distance”  from  the  utility-cost  Pareto  Front  in  a  given  epoch) 
improves  by  executing  a  change  mechanism.  The  FPS  can  range  from  -100  (move  from  frontier 
to  most  dominated)  to  0  (no  change)  to  +100  (move  from  most  dominated  to  frontier).  Failure  is 
recorded  as  a  “-101”  meaning  the  design  becomes  invalid.  “+101”  is  used  to  show  a  design 
moving  from  invalid  to  on  the  frontier.  The  FPS  can  be  viewed  as  a  distribution  across  epochs  by 
strategy,  and  can  be  used  to  compare  shapes  of  the  distribution  between  designs,  as  well  as  in  a 
percentile  summary  table.  FPS  is  calculating  the  magnitude  of  the  value  effects  of  the  “best” 
design  transition  in  each  epoch  and  can  vary  significantly  between  strategies  as  different  rule 
execution  logic  and/or  restrictions  are  imposed,  changing  most  desirable  end  states. 

FPS  Insights  by  strategy: 

Using  the  “maximize  utility”  strategy  (Figure  28),  designs  C,  D,  E,  and  F  are  never 
invalid  when  changeability  is  considered.  Maximizing  utility  generally  has  a  slight  negative 
effect  on  efficiency,  with  the  exception  of  design  F.  Designs  D,  E,  and  G  do  not  execute  changes 
in  a  majority  of  epochs.  Designs  A  and  F  have  the  most  effective  improvements  in  efficiency. 
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Figure  28.  Space  Tug  FPS  Distribution  for  Maximize  Utility  Strategy 


Using  the  “maximize  efficiency”  strategy  (Figure  29),  one  can  see  that  it  does  not  allow 
for  negative  FPS  changes,  excepting  unavoidable  failure. 


Figure  29.  Space  Tug  FPS  Distribution  for  Maximize  Efficiency  Strategy 
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Using  the  “survive”  strategy  (Figure  30),  one  can  see  that  there  are  many  fewer  changes, 
with  the  exception  of  design  A,  which  must  change  always  as  it  will  run  out  of  fuel  if  operated  in 
consecutive  epochs. 


Figure  30.  Space  Tug  FPS  Distribution  for  Survive  Strategy 

In  addition  to  calculating  FPS,  Removal  Weakness  can  be  performed,  which  looks  at  the 
degree  to  which  a  design  depends  on  a  particular  change  mechanism  for  its  valuable 
changeability.  This  information  is  important  for  assessing  the  criticality  of  a  change  mechanism, 
showing  how  valuable  a  system  would  be  if  the  mechanism  failed.  For  this  system,  most  of  the 
change  mechanisms  are  redesign  types,  which  doesn’t  suffer  from  potential  breakdowns. 
Performing  a  removal  weakness  on  rule  6  just  makes  the  DFC  level  2  designs  identical  to  DFC 
level  1  designs,  but  with  an  additional  weight  penalty. 

One  more  analysis  can  be  performed  looked  at  the  Available  Rank  Increase  which 
approximates  value  as  the  number  of  designs  (ranks)  a  design  can  surpass  in  utility  via  change 
mechanisms.  This  is  an  imperfect  metrics  (no  accounting  for  costs  and  affected  heavily  by  design 
enumeration),  but  can  be  an  interesting  basis  for  comparison  of  change  mechanisms  as  utility 
enablers. 

Figure  3 1  illustrates  the  ARI  calculation  across  the  design  space,  comparing  rule  usage. 
The  take-away  is  that  rules  2,  4,  and  6  are  used  the  most  in  increasing  ARI  (e.g.  utility  gain)  and 
these  three  rules  all  relate  to  amount  of  fuel  on-board.  This  highlights  the  important  utility¬ 
enabling  characteristic  of  having  more  fuel  available  to  the  Space  Tug. 
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Figure  31.  Space  Tug  ARI  Comparison  across  Change  Rules 


Step  5:  Conduct  Era  Simulation  and  Analysis 

In  the  step  5,  epochs  are  time  sequenced  to  determine  performance  of  systems  across  a 
lifecycle  and  can  give  insights  into  the  path  dependence  of  rule  execution  and  likelihood  of  using 
change  mechanisms  given  strategies. 

Figure  32  illustrates  a  roll-up  of  the  Era  Analysis  for  Design  E  across  5000  potential  eras. 
For  these  eras,  and  across  the  four  considered  strategies,  only  rules  4  and  5  were  executed.  The 
probabilistic  nature  of  the  results  is  because  the  rule  execution  was  dependent  on  the  particular 
era  (time-sequenced  and  duration-labeled  epochs)  that  unfolded.  One  of  the  key  insights  from 
step  5  is  the  ability  to  generate  a  “going  rate”  for  changeability  against  other  metrics,  such  as 
cost. 


If  we  decide  on  design  E,  then 
we  might  consider  investing  in 
Rules  d  and  5 

Rule  4:  swap  fuel  tank 
Rule  5:  change  capability 


Likelihood  of  Design  E  executing  each  transition  rule 
across  a  10  year  era  (per  strategy) 


Strategy 

Rule  1 

Rule  2 

Rule  3 

/Rule  4 

Rule  $ 

Rule  6 

MaxLI 

N/A 

N/A 

N/A  l 

©100.0% 

0  89.2% 

V  N/A 

MaxEff 

N/A 

N/A 

N/A 

0  100.0% 

097.1% 

N/A 

Survive 

N/A 

N/A 

N/A  \ 

Id  94.9% 

Q  0.0% 

/  N/A 

MaxP 

— 

N/A 

j  96.8% 

031.5°^ 

N/A 

Figure  32.  Space  Tug  Design  E  Rule  Usage  by  Strategy  across  a  10  Year  Era 


As  illustrated  in  Table  18,  by  comparing  designs  with  and  without  change  mechanisms 
enabled,  one  can  determine  the  costs  and  benefits  of  adding  such  changeability  across  the  system 
lifecycle. 
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Table  18.  Space  Tug  Changeability  Lifecycle  Cost/Benefit  Tradeoff 

-DFC  tradeoff  Design  +DFC  tradeoff 


N/A 


D 


-$80M  initial  cost, 

-$4B  profit  over  10  years 

-$384M  initial  cost, 

-$20B  profit  over  10  years 


+$544M  initial  cost, 

+$34B  profit  over  1 0  years 

+$80M  initial  cost, 

+$21 B  profit  over  1 0  years 

N/A 


4.1.4.3  Satellite  Radar  Case  Study 

The  primary  purpose  of  the  application  of  the  Satellite  Radar  case  in  this  research 
investigation  was  to  demonstrate  scalability  of  the  end-to-end  process  in  more  complex  case. 

After  completing  analysis  on  the  X-TOS  and  Space  Tug  data  sets,  the  research  team 
began  the  application  of  VASC  to  a  Satellite  Radar  System  (SRS).  SRS  is  a  satellite 
constellation  designed  to  provide  24-hour  all-weather  imaging  and  tracking  of  strategic  ground 
targets.  The  SRS  data  set  is  much  larger  than  the  previous  two,  featuring  23,328  designs  (from 
12  design  variables),  972  epochs  (from  6  epoch  variables),  and  8  change  mechanisms.  In 
addition  to  this,  the  SRS  model  is  designed  to  track  the  entire  system  lifecycle  through  the 
Design,  Build,  Test,  and  Operations  phases,  with  different  expected  schedule  times  for  each 
design  and  different  change  mechanisms  available  at  different  costs  in  each  of  the  phases. 
Together,  these  features  combine  to  make  for  a  robust  case  study,  but  one  that  is  significantly 
more  challenging  to  implement  in  both  the  multi-epoch  and  era  domains  than  previous  VASC 
cases. 

Although  VASC  can  be  a  time-consuming  process,  with  the  smaller  case  studies 
requiring  about  a  day  of  computation  time  without  including  the  time  for  data  set  creation  or 
multi-arc  transition  generation,  the  main  restriction  for  the  large  case  studies  is  not  time,  but 
computer  memory.  The  single-arc  transition  matrices  for  SRS  (after  being  collapsed  together) 
take  about  20GB  of  RAM  alone,  which  is  more  than  many  workstation  class  computers  are  being 
built  with  today  and  certainly  too  much  for  older  equipment.  This  was  the  main  barrier  for  the 
research  team’s  attempted  application  of  VASC  to  SRS,  as  the  current  MATLAB® 
implementation  of  the  VASC  code  base  was  unable  to  process  such  a  large  amount  of  data.  It  is 
anticipated  that  further  work  to  optimize  the  code  base  to  address  this  limitation  would  enable 
the  research  team  to  conduct  the  analysis,  however  the  research  period  of  performance  ended 
before  this  could  be  accomplished. 

Another  barrier  to  applying  VASC  to  SRS  is  the  newly  included  considerations  of 
schedule  tracking  and  lifecycle  phases.  Conceptually,  these  are  not  irreconcilable  with  VASC. 
Because  lifecycle  phase  determines  which  change  mechanisms  are  available  and  how  much  they 
cost,  the  strategies  must  be  applied  to  each  phase  separately  in  Step  3.  In  some  sense,  this 
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suggests  that  a  multi-phase  study  in  VASC  will  simply  take  the  form  of  VASC  applied 
independently  to  each  phase  in  steps  3  and  4,  with  the  results  intelligently  combined  by  the  team 
to  draw  conclusions.  Era  analysis  remains  the  same,  as  modeling  a  system  lifecycle  by  necessity 
requires  progressing  through  the  phases  together  (not  independently),  but  demands  a  more 
sophisticated  era  constructor  and  simulation  code.  Since  each  design  has  an  expected  duration 
for  each  phase,  when  a  transition  is  executed,  the  time  cost  needs  to  be  applied  as  a  delay  in 
schedule  and  then  the  new  expected  duration  would  be  the  duration  of  the  end  state  minus  the 
original  duration.  Phase  changing  will  occur  independently  of  epoch  switches,  as  soon  as  the 
schedule  time  is  reached.  In  addition  to  providing  a  more  realistic  lifecycle  model,  the  inclusion 
of  phases  allows  for  additional  interesting  strategies  and  era-analysis  metrics.  Because  “time  to 
fielding  the  system”  is  typically  an  important  design  criterion,  “minimize  expected  schedule 
time”  is  a  potential  strategy  of  interest;  additionally  total  schedule  delay  (transition  cost  plus 
potential  extended  schedule  time  of  new  design)  can  be  used  as  a  threshold,  as  in  “maximize 
utility  without  increasing  expected  schedule  time  by  more  than  1  year.”  Average  total  time  to 
fielding  including  all  transitions  can  be  an  output  of  era  analysis  as  well. 

The  research  team’s  inability  to  apply  VASC  to  SRS  is  a  reflection  of  the  ambitious 
scope  of  the  research,  rather  than  infeasibility  of  VASC  itself.  The  team  anticipated  challenges 
regarding  the  scalability  and  complexity  involved  in  application  of  VASC  to  this  case,  and  in 
attempting  to  apply  VASC,  was  able  to  recognize  the  potential  insights  that  could  be  generated 
on  such  a  case,  as  well  as  the  challenges  needed  to  be  overcome  in  maturing  VASC  for  general 
applicability.  These  challenges  are  further  discussed  in  the  Discussion  section  below. 

4.1.5  Publications 

During  the  performance  of  the  contract,  one  conference  publication  was  published  and 
presented  at  the  9th  Conference  on  Systems  Engineering  Research  during  April  2011  in  Los 
Angeles,  CA.  The  paper  "A  Method  Using  Epoch-Era  Analysis  to  Identify  Valuable 
Changeability  in  System  Design,"  and  was  authored  Matthew  Fitzgerald,  Adam  Ross,  and  Donna 
Rhodes.  The  paper  is  available  in  the  proceedings  and  on  the  MIT  SEAri  website 
(http://seari.mit.edu).  The  research  team  expects  to  publish  one  additional  paper  in  Spring  2012. 
In  addition,  a  masters  thesis  related  to  the  project  will  be  forthcoming  in  May  201 1  by  graduate 
student  Matthew  Fitzgerald,  and  following  publication  will  be  posted  on  the  SEAri  website. 

4.2  Discussion 

4.2.1  Fit  of  Research  Outcomes  with  Larger  META  Projects 

In  an  effort  to  fit  the  changeability  (a.k.a.  “metrics  for  adaptability”)  into  the  larger 
META  project,  the  research  team  participated  in  a  metrics  workshop  at  the  July  PI  meeting. 
During  this  workshop,  a  notional  figure  of  META  design  flow  was  shared.  Figure  33  below 
indicates  where  in  the  META  design  flow  the  VASC  work  positions  itself.  It  is  currently 
envisioned  that  VASC  would  be  performed  as  part  of  the  tradespace  evaluation  step,  following 
tradespace  enumeration  and  model  generation  in  order  to  evaluate  the  tradespace.  Since  VASC 
requires  performance  of  alternative  designs  to  be  evaluated  across  a  number  of  epochs  (context- 
need  pairs),  one  must  have  performance  models  that  can  generate  this  data.  Additionally,  the 
META  tools  must  add  in  considerations  for  change  mechanisms  and  path  enablers.  This 
consideration  is  in  addition  to  the  classical  design-performance  considerations  shown  implicitly 
in  the  design  flow  picture. 
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Figure  33.  META  Design  Flow  with  Changeability  Metrics  Indicated 
4.2.2  Scalability  of  Approach 


As  seen  in 


Table  19,  the  VASC  has  mostly  friendly  scaling  properties  to  extremely  large  projects, 
with  the  exception  of  collapsing  multi-arc  transitions  (“Rule  Collapse”),  which  scales  poorly 
with  number  of  change  mechanisms.  A  simple  workaround  at  this  point  is  to  only  consider 
“single-arc  transition”,  meaning  execution  of  one  change  mechanism  per  epoch.  The  approach  is 
valid,  though  less  informative,  when  considering  such  single-arc  transitions.  It  is  expected  that 
additional  research  will  be  able  to  uncover  appropriate  approximations  or  alternative  algorithms 
that  can  improve  the  approximate  scaling  order  of  Rule  Collapse. 
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Table  19.  Scalability  of  Activities  in  VASC 


Activity 

Worst  Variable  to 
Increase 

Approximate  Order 

Epoch  Dataset  Generation 

#  model  runs/context 

Linear,  but  model  runtime  is  critical 

Transition  Matrices  Generation 

#  designs 

N2  (worst  case,  depends  on  rule  algorithm) 

Rule  Collapse 

#  change  mechanisms 

Factorial 

FPN 

#  designs 

Linear,  but  depends  on  shape  of 
tradespace 

NPT  (fuzzy,  effective) 

#  designs  /  epochs 

Linear  in  both 

Strategy  End  State  Calculation 

#  designs 

Linear,  but  time  depends  on  complexity  of 
strategy 

FPS 

#  designs 

Simple  arithmetic,  but  depends  on  having 
FPN  and  Strategy  calculated 

Removal  Weakness 

change  mechanisms  -  1 

Requires  a  repeat  of  Ruie  Collapse 

Era  Simulation 

#  eras  per  design  /  strategy 
complexity 

Linear,  but  dependent  on  strategy  if  some 
era  information  is  known 

An  additional  concern  regarding  scalability  is  the  execution  time  associated  with  VASC. 
As  little  time  was  spent  on  code  optimization,  speeding  up  current  algorithm  runtime  is  certainly 
possible  through  additional  research.  Related  to  execution  time,  the  research  team  uncovered  a 
number  of  programming  challenges  when  attempting  to  apply  VASC  to  larger  data  sets,  which  is 
described  briefly  in  the  following  section. 

4.2.2. 1  Programming  Challenges  Uncovered  in  SRS  Case  Study 

The  SRS  data  set  is  much  larger  than  the  other  two  data  sets  used  in  this  research, 
featuring  23,328  designs  (from  12  design  variables),  972  epochs  (from  6  epoch  variables),  and  8 
change  mechanisms.  VASC  is  feasible  for  case  studies  as  large  as,  or  larger  than,  SRS.  What 
will  be  key  in  implementing  VASC  on  such  cases  is  intelligent  use  of  data  management  and 
parallelization.  MATLAB®  stores  all  of  the  variables  in  the  workspace  in  active  RAM  and, 
when  the  amount  of  data  in  the  workspace  nears  the  maximum  limit,  all  operations  begin  to  slow 
down  exponentially.  Ideally,  only  the  data  needed  at  any  given  time  would  be  in  the  workspace. 
Since  loading  and  unloading  data  from  the  workspace  takes  a  finite  amount  of  time  directly 
proportional  to  the  amount  of  data,  there  will  be  some  optimal  amount  of  data  to  be  stored  and 
called  at  a  time.  For  example,  during  era  simulation  it  is  unnecessary  to  have  access  to  the 
strategic  changes  for  any  design  but  the  current  design;  a  code  structure  that  loads  only  the 
relevant  changes  and  swaps  them  out  for  the  appropriate  set  whenever  the  system  transitions  to 
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another  design  would  minimize  memory  requirements  but  require  a  load/unload  of  data  after 
every  transition.  The  current  method  (all  data  stored  as  active)  maximizes  memory  requirements 
and  would  minimize  time  requirements  if  not  for  the  slowdown  associated  with  maxing  out  a 
computer’s  RAM.  Thus,  an  optimal  point  for  the  time  to  run  VASC  lies  between  these  two 
extremes,  loading  and  unloading  small  batches  of  data  as  they  become  necessary.  The  research 
team  at  SEAri  will  be  looking  into  this  modification  in  the  near  future,  but  it  will  likely  require 
an  overhaul  of  much  of  the  existing  VASC  code  base.  The  research  team  is  optimistic  that 
experienced  programmers  who  seek  to  mature  the  VASC  code  base  will  be  able  to  make  these 
improvements  and  overcome  limitations  inherent  in  the  MATLAB®  code  base  developed  in  this 
research  task  by  less  experienced  student  programmers. 

The  other  key  potential  improvement  available  to  reduce  the  runtime  of  VASC  is 
parallelization:  tasking  multiple  computer/processors  with  different  jobs  simultaneously  to 
accomplish  more  work  at  once.  VASC  is  well  suited  to  parallelization  in  all  of  its  steps,  as  the 
calculations  required  are  mostly  independent  across  designs  and  epochs.  As  an  example, 
consider  the  determination  of  strategic  transitions  in  Step  3:  the  logic  used  to  determine  if  and 
how  a  given  design  will  transition  in  a  given  epoch  is  completely  independent  of  all  other 
determinations  of  the  same  type,  thus  allowing  the  process  to  be  split  up  amongst  any  number  of 
available  computers  with  only  the  small  additional  cost  of  eventually  combining  all  of  the  results 
together.  This  is  also  true  for  multi-epoch  analysis  (each  design  can  be  evaluated  separately)  and 
era  analysis  (each  simulated  era  is  independent  of  all  others).  The  existing  VASC  code  base  is 
not  equipped  to  handle  parallelization  right  now,  but  implementing  it  is  a  less  intrusive  change 
than  the  active  workspace  modification  from  the  previous  paragraph,  as  it  simply  requires 
specifying  which  designs/epochs  should  be  evaluated  in  each  script  rather  than  requesting  all  of 
them  at  once.  While  the  active  workspace  change  will  allow  less  powerful  computers  to  perform 
VASC,  the  parallelization  technique  will  be  more  critical  in  speeding  up  the  total  computational 
time.  Because  of  the  high  degree  of  independence  between  tasks,  two  computers  will  do  the 
computations  in  about  half  the  time  and  so  on,  with  diminishing  returns  only  appearing  once  the 
number  of  computers  exceeds  the  combinations  of  designs/epochs  or  designs/eras  in 
consideration  (easily  in  the  millions  for  even  moderately  sized  studies). 

4.2.3  Incorporating  into  Existing  Studies 

The  following  are  general  guidelines  for  the  types  of  data  needed  to  evaluate 
changeability  using  VASC: 

•  Data  to  characterize  design  alternatives  (design  variables  and  attributes=decision 
criteria  that  differentiate  alternatives,  such  as  performance) 

•  Clearly  defined  context  variables  affecting  perceived  system  value 

o  Variables  need  to  differentiate  epochs 

o  Must  affect  value  in  a  significant/meaningful  way  for  useful  information 
to  exist  beyond  a  single  epoch 

•  Change  mechanism  with  execution  cost  data 

•  Requisite  information  for  chosen  changeability  value  metric 

o  Value  must  be  able  to  be  calculated  for  each  epoch  (e.g.  “mission  utility) 

In  order  to  incorporate  VASC  into  existing  studies,  one  must  make  the  following 
considerations: 
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One  must  have  a  means  for  selecting  the  set  of  “designs  of  interest”  for  valuable 
changeability  analysis.  In  the  examples  shown  in  this  report,  screening  metrics  were  used, 
however  one  can  use  whatever  means  makes  sense  for  a  particular  study. 

One  must  also  have  a  means  for  choosing  the  value  metric  for  what  is  considered 
“valuable”  about  changeability.  Depending  on  the  study,  this  choice  may  change.  For  example, 
for  commercial  systems,  impact  on  net  profit  may  make  sense,  while  for  military  systems,  impact 
on  net  utility  may  make  sense.  In  VASC,  a  combination  of  rule  execution  strategy  and  “going 
rate”  captures  the  long  run  value  definition  (e.g.  accumulated  mission  utility  vs.  cost  of 
investment  in  the  changeability). 

One  must  also  have  a  means  for  transforming  the  design  performance  data  set  into  an 
epoch-differentiated  data  set.  Any  form  of  discrete  context  separation  should  work.  For 
example,  one  can  perform  sensitivity  analysis  on  the  performance  model,  with  each  distinct  level 
of  the  varied  parameters  representing  a  different  possible  future  context.  This  makes  particular 
sense  when  one  is  varying  the  uncertain  assumed  constants  in  a  model.  Other  example  variations 
that  could  be  performed  to  generate  epochs  include:  varying  stakeholder  preferences  (including 
required  performance  levels  or  thresholds),  varying  uncertain  physical  parameters  (e.g.,  drag 
coefficient)  or  cyclical  variables  (e.g.,  solar  activity),  or  varying  underlying  constants  in  a  Real 
Options/NPV  analysis  (e.g.,  growth  spread,  volatility,  risk  free  rate,  discount  rate). 

4.2.4  Change  Mechanisms  Elaborated 

One  of  the  principle  constructs  used  in  this  research  is  the  idea  of  a  “change  mechanism,” 
which  is  the  means  by  which  a  system  design  changes  state.  Interpretation  of  what  is  considered 
a  “design  state  change”  is  subject  to  discussion  and  must  be  appropriately  defined  for  the  level  of 
analysis  considered.  In  this  research,  the  level  of  analysis  is  at  the  conceptual  design  level,  so  a 
state  change  was  interpreted  in  terms  of  changes  between  alternative  designs  under 
consideration.  Figure  34  illustrates  where  the  concept  of  “change  mechanism”  fits  within  a  larger 
set  of  constructs  related  to  design  and  changeability  valuation.  The  green  box  indicates  the  Path 
Enabler-Change  Mechanism  pair  as  a  “change  option,”  that  is,  by  having  path  enablers,  one  has 
the  option  to  execute  a  change  mechanism  in  order  to  change  the  system  choice  from  one  state  to 
another.  Following  change  mechanisms,  which  are  implemented  in  tradespace  network  analysis 
as  “change  rules,”  one  can  begin  to  perform  changeability  analysis  to  assess  the  “degree”  to 
which  a  design  has  changeability  (the  counting  aspect  of  changeability  addressed  in  this 
research).  Following  this,  one  can  perform  alternative  analysis  techniques  (such  as  Real  Options 
Analysis,  or  Epoch-Era  Analysis)  in  order  to  determine  the  “value”  of  the  changeability  (the 
magnitude  aspect  of  changeability  addressed  in  this  research). 


Figure  34.  Change  Mechanisms  as  Part  of  Change  Options 
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In  order  to  help  illustrate  this  set  of  constructs,  Table  20  lists  several  systems  with  their  path 
enablers  and  change  mechanisms,  along  with  a  rough  order  of  potential  end  states  through  the 
listed  change  options.  Table  21  provides  additional  change  mechanism  examples. 


Table  20.  Examples  of  Systems,  Path  Enablers,  and  Change  Mechanism 


System 

Path  Enabler 

Change  Mechanism 

Potential 
End  States 

ARAM  IS  satellite 
architecture 

Modular  tile  components 

R  eco  nfiguri  ng/a  d  d  i  ng/s  u  bstit  uti  n  g  ti  les 

Many 

iPhone 

Mobile  App  Store  and  extensible 
software  architecture 

Adding  new  function 

Many 

Space  station 
w/shuttle 

Docking  w/  extra  fuel  and 
thrusters 

Increasing  ISS  altitude  by 
shuttle  firing  thrusters 

Many 

F-14 

Mechanical  hinged  wings 

Changing  wing  sweep  angle 

Few 

Atlas  V 

Strap-on  solid  rocket  motors 

Scaling  up  or  down  lift  capability 

Few 

US  ISRSoS 

Reprogrammable  Global  Hawk 
flight-path  waypoints 

Changing  surveillance  area,  as 
reeded,  by  combatant  commander 

Infinite 

Table  21.  Additional  Example  Change  Mechanisms 


System 

State  Change 

Mechanism 

Agent 

Convertible 

Closed-roof  to 
open-roof 

Button-activated  automatic 
mechanism 

Internal  driver 

Military  Aircraft 

Adding  external 
fuel  tanks 

Universal  fittings  for 
tanks/missiles/etc 

External  crew 

V-22  Osprey 

Vertical  to  horizontal 
flight  modes 

Rotating  mechanism  on 
rotors 

Internal  pilot 

Parachute 

Packed  to  deployed 

Pulling  release  pin  to 
deploy  pilot  chute 

External  parachutist 

Parachute 

Packed  to  deployed 

Explosive  activated  to  pull 
release  pin 

Internal  emergency 
AAD  pull  device 

HVAC  system 

Change  temperature 

Activating  heater  or  air 
conditioner 

Control  thermometer 

BMW  production 
plant 

Change  in  car 
order  configuration 

Build-your-own  vehicle 
online  service 

External  buyer 

BMW  production 
plant 

Change  between  5,  6, 
or  7  series  production 

Similar  joining  sequences 
and  load  carrying  points  for 
assembly 

Factory  management 
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4.2.5  Considerations  for  Implementation 

One  of  the  benefits  of  VASC,  is  that  it  can  accelerate  and  focus  attention  on  essential 
aspects  of  the  valuable  changeability  analysis  through  targeted  software  automation.  Figure  35 
illustrates  which  parts  of  VASC  require  human  input,  which  are  software-automated,  and  which 
require  human  interpretation.  VASC  was  intentionally  developed  in  order  to  leverage  automation 
to  the  fullest  extent  possible,  and  to  explicitly  highlight  when  human  expert  input  is  required,  due 
to  intrinsically  subjective  aspects  of  judgment  related  to  definitions  of  what  is  considered 
“valuable.” 


Figure  35.  Human  vs.  Automation  in  Approach 


In  VASC,  humans  must  define  the  following: 

•  Possible  change  mechanisms 

•  “Rules”  as  algorithmic  implementation  of  mechanisms 

•  Possible  design  features  that  enable  change  mechanisms  (“Path  Enablers”) 

•  Possible  rule  execution  strategies 

•  Possible  epochs  (contexts  and  need  pairs  encountered  by  the  system) 

•  Possible  design  space  (system  alternatives  to  be  analyzed) 

Given  these  inputs,  software  conducts  much  of  the  analysis,  providing  a  set  of  changeability 
metrics  and  distributions  for  human  interpretation  at  the  end.  In  particular  these  metrics  seek  to 
give  cost  and  benefit  insights  into  when,  where,  and  why  changeability  may  provide  value.  The 
VASC  also  allows  decision  makers  to  construct  and  compare  alternative  strategies  and  design 
choices  that  enable  short  run  and  long  run  changeability,  in  an  iterative  fashion,  promoting  up 
front  consideration  of  changeability  as  a  value-enhancing  set  of  decisions. 

4.2.6  Future  Research 

Future  research  on  VASC  will  attempt  to  solve  the  challenges  encountered  when 
applying  the  approach  to  the  Satellite  Radar  System  (SRS)  case  study,  including  scalability  of 
the  metric  calculation  algorithms.  Insights  from  lifecycle-phase  dependent  analysis  will  be 
generated  and  are  anticipated  to  provide  high-level  guidance  into  investment  in  path  enablers  as  a 
function  of  desired  lifecycle  phase  characteristics.  In  particular,  the  ultimate  aim  of 
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“adaptability”  as  inferred  from  the  DARPA  META  program  is  being  able  to  respond  to  a 
perturbation  on  appropriate  timescales.  This  means  that  changeability  should  “match”  response 
time  in  the  system  to  the  perturbations  encountered  by  the  system  throughout  its  lifecycle. 
Change  mechanisms  that  act  within  particular  lifecycle  phases  will  have  more  or  less  importance 
relative  to  particular  perturbations.  Executed  change  mechanisms  may  also  send  a  system 
backwards  in  the  lifecycle  if  such  changes  result  in  a  need  to  “re-test”  or  even  “rebuild”  the 
system  (see  Figure  36).  Future  work  will  consider  this  lifecycle  aspect  explicitly  and  seek  to 
build  analysis  and  guidance  on  which  mechanism  to  invest  based  on  expected  (and  possible 
unexpected)  perturbations  types.  The  farther  a  change  goes  back  into  the  lifecycle,  the  longer 
(usually)  it  takes  before  utility  is  experienced  again.  Choices  can  be  made  to  give  an  option  to 
change  later  in  the  lifecycle,  or  to  reduce  the  time  and  cost  for  getting  back  to  operations.  In  this 
way,  changeability  can  be  embedded  in  a  manufacturing  capability  (e.g.,  DARPA  iFab)  or  in  the 
system  itself.  VASC  will  seek  to  enable  the  comparison  of  tradeoff  of  both  of  these  types  of 
changeability  within  a  common  framework. 
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Figure  36.  Delay  in  Experienced  Utility  as  a  Function  of  Change  Mechanism- 

Lifecycle  Phase 
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5.  CONCLUSIONS 

The  primary  goal  for  the  research  was  to  uncover  difficult  to  extract  information  on 
valuable  changeability  for  a  design  space  and  present  it  in  an  accessible  way  to  assist  in  decision 
making.  Other  important  goals  included  identifying  designs  which  deliver  high  amounts  of  value 
in  different  ways  (e.g.  robustness  and  changeability),  and  the  operational  strategies  that 
maximize  value.  The  research  also  enabled  the  assessment  of  what  change  mechanisms  deliver 
the  most  value  or  are  the  most  critical  for  some  designs  to  continue  to  deliver  value  over  time. 
Ultimately,  in  order  to  help  to  justify  the  investment  in  changeability,  which  has  been  difficult  to 
do  in  the  past  due  to  asymmetry  in  ease  of  identifying  costs  over  benefits,  the  research  has 
demonstrated  a  first  effort  in  being  able  to  establish  a  cost  vs.  benefit  tradeoff  for  adding  or 
removing  changeability  from  a  design  (a.k.a.  the  “going  rate”  for  changeability). 

In  order  to  organize  and  assist  analysts  and  decision  makers  in  capturing  changeability 
tradeoffs  within  a  study,  the  Valuation  Approach  for  Strategic  Changeability  (VASC)  was 
developed.  VASC  is  a  five  step  approach  that  guides  analysts  through  generation  and 
organization  of  design  data,  as  well  as  application  of  analysis  to  generate  valuable  changeability 
metrics  and  their  interpretation.  Figure  37  shows  the  high  level  flow  of  data  in  order  to  generate 
the  changeability  metrics  developed  in  this  research. 


Figure  37.  Data  Flow  for  VASC  Metrics 
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5.1  Research  Contributions 

In  addition  to  developing  the  Valuation  Approach  for  Strategic  Changeability  (VASC), 
the  contributions  of  this  research  include: 

•  Expanded  set  of  screening  and  valuation  metrics  (eNPT,  efNPT,  FPN,  FPS)  in 
Table  22 

•  Explicit  method  for  accounting  for  value  of  changeability  over  short  and  long 
time  scales  (strategy-interpreted) 

•  Finked  explicit  design  decisions  with  changeability  (change  rule  comparison) 

•  Incremental  analysis  approach  that  can  scale  with  available  information  and  effort 

•  An  approach  that  is  mostly  automated,  but  also  encourages  focused  value- 
elicitation  and  interpretation  discussions  between  decision  makers  and  analysts 


Table  22.  Final  Set  of  Valuable  Changeability  Metrics 


Aspect  of 
Valuable 
Changeability 

Acronym 

Stands  For 

Definition 

Robustness  via 
“no  change” 

NPT 

Normalized  Pareto  Trace 

%  epochs  for  which  design  is  Pareto 
efficient  in  utility/cost 

Robustness  via 
“no  change” 

fNPT 

Fuzzy  Normalized  Pareto 
Trace 

Above,  with  margin  from  Pareto 
front  allowed 

Robustness  via 
“change” 

eNPT, 

efNPT 

Effective  (Fuzzy) 
Normalized  Pareto  Trace 

Above,  considering  the  design’s  end 
state  after  transitioning 

“Value”  gap 

FPN 

Fuzzy  Pareto  Number 

%  margin  needed  to  include  design 
in  the  fuzzy  Pareto  front 

“Value”  of  a 
change 

FPS 

Fuzzy  Pareto  Shift 

Difference  in  FPN  before  and  after 
transition 

“Value”  of  a 
change 

ARI 

Available  Rank  Increase 

#  of  designs  able  to  be  passed  in 
utility  via  best  possible  change 

Degree  of 
changeability 

OD 

Outdegree 

#  outgoing  transition  arcs  from  a 
design 

Degree  of 
changeability 

FOD 

Filtered  Outdegree 

Above,  considering  only  arcs  below 
a  chosen  cost  threshold 
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APPENDIX  A  -  Publications  and  Presentations 

The  publications  generated  in  this  research  are  available  on  the  SEAri  website: 

Fitzgerald,  M.E.,  Ross,  A.M.,  and  Rhodes,  D.H.,  "A  Method  Using  Epoch-Era  Analysis  to 
Identify  Valuable  Changeability  in  System  Design,"  9th  Conference  on  Systems 
Engineering  Research,  Los  Angeles,  CA,  April  2011. 
http://seari.mit.edu/documents/preprints/FITZGERALD  CSER1  l.pdf 

Presentations  are  available  on  the  DARPA  META  Sharepoint  site. 
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LIST  OF  SYMBOLS,  ABBREVIATIONS,  AND  ACRONYMS 


Organization 

ESD 

Engineering  Systems  Division 

SEAri 

Systems  Engineering  Advancement  Research  Initiative 

Projects 

A-TOS 

A-iteration  Terrestrial  Observer  Swarm 

B-TOS 

B-iteration  Terrestrial  Observer  Swarm 

C-TOS 

C-iteration  Terrestrial  Observer  Satellite 

X-TOS 

X-iteration  Terrestrial  Observer  System 

SR 

Satellite  Radar 

TPF 

Terrestrial  Planet  Finder 

Process/Methods 

DFC 

Design  for  Changeability 

EEA 

Epoch-Era  Analysis 

MATE 

Multi-Attribute  Tradespace  Exploration 

MAUA 

Multi- Attribute  Utility  Analysis 

MAUF 

Multi- Attribute  Utility  Function 

VASC 

Valuation  Approach  for  Strategic  Changeability 

Metrics 

ARI 

Available  Rank  Increase 

eNPT 

Effective  Normalized  Pareto  Trace 

efNPT 

Effective  Fuzzy  Normalized  Pareto  Trace 

fNPT 

Fuzzy  Normalized  Pareto  Trace 

FOD 

Filtered  Outdegree 

FPN 

Fuzzy  Pareto  Number 

FPS 

Fuzzy  Pareto  Shift 

NPT 

Normalized  Pareto  Trace 

OD 

Outdegree 

Variables 

DM 

Decision  Maker 

DV 

Design  Variable 

MOE 

Measure  of  Effectiveness 

TPM 

Technical  Performance  Measure 

X 

Attribute 

u 

Utility 
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GLOSSARY  OF  TERMINOLOGY 

Adaptability  (DARPA):  the  ability  of  a  system  to  change  easily,  quickly,  and 
inexpensively  (i.e.,  with  minimum  incurrence  of  cost  and  degradation  in  performance)  in 
response  to  a  wide  spectrum  of  anticipated  and  unanticipated  perturbation  events  exogenous  or 
endogenous  to  the  system. 

Adaptability  (MIT  SEAri):  ability  of  a  system  to  be  changed  by  a  system- internal 
change  agent  with  intent 

Changeability:  ability  of  a  system  to  alter  its  form  or  operations,  and  consequently 
possibly  its  function,  at  an  acceptable  level  of  resources 

Change  Mechanism.  A  method  by  which  the  system  is  changed.  An  example:  “Bum  on 
board  fuel”  results  in  change  in  satellite  orbit,  costing  “extra  ops  cost”  for  executing  the 
maneuver  (system  “state”  includes  operating  orbit). 

Change  Rule.  An  algorithm  that  determines  whether  two  proposed  “states”  are 
connected  through  a  particular  change  mechanism.  An  example:  “Compare  two  ‘states’  and  if 
difference  is  only  fuel  and  orbit  location,  then  if  fuel  difference  is  equal  to  amount  burned  to 
achieve  orbit  difference,  then  states  have  directed  accessibility  via  change  mechanism  for  cost 
determined  by  that  mechanism.  The  change  rale  is  an  operationalization  of  the  concept  of  change 
mechanism  in  order  to  allow  for  computationally  generated  and  evaluated  alternative  “paths” 

Epoch:  An  Epoch  is  a  period  for  which  the  system  context  has  constant  value 
expectations.  Each  fixed  context  is  characterized  by  static  constraints,  available  design  concepts, 
available  technology,  and  articulated  attributes 

Era:  An  time-ordered  sequence  of  epochs. 

Filtered  Outdegree.  The  number  of  outgoing  arcs  (change  paths)  from  one  design  at 
acceptable  “cost”  as  a  measure  of  changeability. 

Flexibility  (MIT  SEAri):  ability  of  a  system  to  be  changed  by  a  system-external  change 
agent  with  intent 

Pareto  Set:  characterizes  those  “non-dominated”  designs  of  highest  utility  at  a  given 
cost,  across  all  costs,  or  those  of  lowest  cost  at  a  given  utility,  across  all  utilities 

Real  Option:  A  “real  option”  gives  the  decision  maker  the  right,  but  not  the  obligation, 
to  exercise  an  action  or  decision  at  a  later  point  in  time. 

Tradespace  Network:  a  tradespace  represented  as  a  network,  where  the  nodes  are 
designs  and  the  arcs  are  transition  paths  from  one  design  to  another. 
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